Audio encoders and decoders with program loudness and boundary metadata

Abstract

The present invention provides an audio processing unit and method for decoding a bitstream, as well as a program and a non-temporary medium. [Solution] In the audio processing unit, the decoder 200 includes a buffer memory 201 that stores at least one frame of a bitstream, a state validity checker 203 that performs adaptive loudness processing of the audio data of an audio program indicated by the bitstream or authentication and / or validity check of the metadata and / or audio data of such an audio program, and an audio decoder 202 that decodes the bitstream and generates decoded audio data.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Patent Application No. 61 / 754,882, filed on 21 January 2013, and U.S. Provisional Patent Application No. 61 / 824,010, filed on 16 May 2013. Each application is incorporated herein by reference in its entirety.

[0002] Technical field The present invention relates to the encoding and decoding of an audio data bitstream, which has metadata indicating the loudness processing state of the audio content and the location of the audio program boundary indicated by the bitstream, for audio signal processing. Some embodiments of the present invention generate or decode audio data that is in one of the formats known as AC-3, Enhanced AC-3, or E-AC-3 or Dolby E. [Background technology]

[0003] Dolby, Dolby Digital, Dolby Digital Plus, and Dolby E are trademarks of Dolby Laboratories Licensing Corporation. Dolby Laboratories provides its own implementations of AC-3 and E-AC-3, known as Dolby Digital and Dolby Digital Plus, respectively.

[0004] Audio data processing units typically operate in a blind manner, paying no attention to the processing history of audio data performed before the data is received. This may work in a processing framework where a single entity performs all audio data processing and encoding for various target media rendering devices, and the target media rendering devices perform all decoding and rendering of the encoded audio data. However, this blind processing does not work well (or does not work at all) in situations where multiple audio processing units are distributed or cascaded (chained) across a diverse network and are expected to perform each type of audio processing optimally. For example, some audio data may be encoded for high-performance media systems and need to be converted to a reduced form suitable for mobile devices along the media processing chain. Thus, an audio processing unit may unnecessarily perform a type of processing on the audio data that has already been performed. For example, a volume leveling unit will perform processing on an input audio clip regardless of whether the same or similar volume leveling has been performed on that input audio clip before. As a result, the volume leveling unit may perform leveling even when it is not necessary. This unnecessary processing can cause degradation and / or removal of certain features when rendering the content in the audio data.

[0005] A typical stream of audio data includes both audio content (e.g., audio content for one or more channels) and metadata that describes at least one characteristic of that audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters that are specifically intended to be used to alter the sound of a program delivered to a listening environment. One such metadata parameter is the DIALNORM parameter, which is intended to indicate the average level of dialogue appearing in an audio program and is used to determine the audio playback signal level.

[0006] During playback of a bitstream containing a sequence of various audio program segments (each with a different DIALNORM parameter), the AC-3 decoder uses the DIALNORM parameter of each segment to perform a type of loudness processing that modifies the playback level or loudness so that the perceived loudness of the dialogue in that segment sequence is at a consistent level. Each encoded audio segment (item) in a sequence of encoded audio items (generally) has a different DIALNORM parameter, and the decoder scales the level of each item so that the playback level or loudness of the dialogue for each item is the same or very similar. However, this may require applying different amounts of gain to different items during playback.

[0007] DIALNORM is typically set by the user and is not automatically generated. However, if no value is set by the user, there is a default DIALNORM value. For example, a content creator may use an external device to measure loudness with the AC-3 encoder and then send the results (showing the loudness of the spoken dialogue in the audio program) to the encoder to set the DIALNORM value. Thus, the correct setting of the DIALNORM parameter relies on the content creator.

[0008] There are several different reasons why the DIALNORM parameter in an AC-3 bitstream might be incorrect. Firstly, each AC-3 encoder has a default DIALNORM value that is used during bitstream generation if the DIALNORM value is not set by the content creator. This default value may be substantially different from the actual dialogue loudness level of the audio. Secondly, even if the content creator measures the loudness and sets the DIALNORM value appropriately, a loudness measurement algorithm or meter that does not conform to the recommended AC-3 loudness measurement method may have been used, resulting in an incorrect DIALNORM value. Thirdly, even if the AC-3 bitstream is generated with a DIALNORM value measured and correctly set by the content creator, it may have been altered to an incorrect value during the transmission and / or storage of the bitstream. For example, in television broadcast applications, it is not uncommon for AC-3 bitstreams to be decoded, modified, and then encoded with incorrect DIALNORM metadata information. Thus, the DIALNORM value included in the AC-3 bitstream may be incorrect or inaccurate, and therefore may have a negative impact on the quality of the listening experience.

[0009] Furthermore, the DIALNORM parameter does not indicate the loudness processing status of the corresponding audio data (e.g., what type(s) of loudness processing was performed on the audio data). Until the present invention, audio bitstreams did not include metadata indicating the loudness processing status of the audio content of the audio bitstream (e.g., the type(s) of loudness processing applied to it) or the loudness processing status and loudness of the audio content of the bitstream in a format of the type described herein. Loudness processing metadata in such a format is useful to facilitate the verification of the effectiveness of adaptive loudness processing of an audio bitstream and / or the loudness processing status and loudness of the audio content, particularly in an efficient manner.

[0010] The present invention is not limited to use with AC-3 bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, but for convenience, it is described in embodiments that generate, decode, or otherwise process such bitstreams including loudness processing status metadata.

[0011] An AC-3 encoded bitstream contains metadata and audio content with one to six channels. The audio content is audio data compressed using perceptual audio coding. The metadata includes several audio metadata parameters intended for use in altering the sound of the program delivered to the listening environment.

[0012] The details of AC-3 (also known as Dolby Digital) encoding are well known and have been described in numerous publications, including Non-Patent Document 1, Patent Documents 1, 2, 3, 4, and 5. All of this is incorporated here in its entirety by reference.

[0013] Details of Dolby Digital Plus (E-AC-3) are described in Non-Patent Document 2.

[0014] Details of Dolby E encoding are described in Non-Patent Documents 3 and 4.

[0015] Each frame of an AC-3 encoded audio bitstream contains audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48kHz, this represents 32 milliseconds of digital audio or 31.25 frames per second of audio.

[0016] Each frame of an E-AC-3 encoded audio bitstream contains audio content and metadata for 256, 512, 768, or 1536 samples of digital audio, depending on whether the audio data contained in the frame is in one, two, three, or six blocks, respectively. For a sampling rate of 48 kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or 189.9, 93.75, 62.5, or 31.25 frames per second of audio, respectively.

[0017] As shown in Figure 4, each AC-3 frame is divided into sections (segments). The sections include: a synchronization information (SI) section containing the synchronization word (SW) and the first of two error correction words (CRC1) (as shown in Figure 5); a bitstream information (BSI) section containing most of the metadata; six audio blocks (AB0 to AB5) containing the data-compressed audio content (and may also contain metadata); a waste bit segment (W) containing any unused bits remaining after the audio content has been compressed; an auxiliary (AUX) information section which may contain further metadata; and the second of two error correction words (CRC2). The waste bit segment (W) is sometimes referred to as a “skip field”.

[0018] As shown in Figure 7, each E-AC-3 frame is divided into sections (segments). The sections include: a synchronization information (SI) section containing the synchronization word (SW) (as shown in Figure 5); a bitstream information (BSI) section containing most of the metadata; one to six audio blocks (AB0 to AB5) containing the data-compressed audio content (and may also contain metadata); a waste bit segment (W) containing any unused bits remaining after the audio content has been compressed (only one waste bit segment is shown, but typically there is a different waste bit segment following each audio block); an auxiliary (AUX) information section which may contain further metadata; and a error correction word (CRC). The waste bit segment (W) is sometimes referred to as a “skip field”.

[0019] AC-3 (or E-AC-3) bitstreams have several audio metadata parameters specifically intended for use in altering the sound of a program delivered to the listening environment. One such metadata parameter is the DIALNORM parameter, which is included in the BSI segment.

[0020] As shown in Figure 6, the BSI segment of an AC-3 frame includes a five-bit parameter ("DIALNORM") indicating the DIALNORM value for the program. If the audio encoding mode ("acmod") of the AC-3 frame is "0", indicating that a dual-mono or "1+1" channel configuration is being used, then a five-bit parameter ("DIALNORM2") indicating the DIALNORM value for a second audio program carried in the same AC-3 frame is included.

[0021] The BSI segment includes a flag ("addbsie") that indicates the presence (or absence) of additional bitstream - information following the "addbsie" bit, a parameter ("addbsil") that indicates the length if there is additional bitstream information following the "addbsil" value, and additional bitstream information up to 64 bits ("addbsi") following the "addbsil" value.

[0022] The BSI segment includes other metadata values not specifically shown in FIG. 6.

Prior - art documents

Patent documents

[0023]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Non - patent documents

[0024]

Non - patent Document 1

Non - patent Document 2

[0025] In some embodiments of a class of the present invention, the present invention is an audio processing unit comprising a buffer memory, an audio decoder, and a parser. The buffer memory stores at least one frame of an encoded audio bitstream. The encoded audio bitstream comprises audio data and a metadata container. The metadata container comprises a header, one or more metadata payloads, and protection data. The header includes a synchronization word that identifies the beginning of the container. The one or more metadata payloads describe an audio program associated with the audio data. The protection data is located after the one or more metadata payloads. The protection data may also be used to verify the integrity of the metadata container and the one or more payloads within the metadata container. The audio decoder is coupled to the buffer memory and can decode the audio data. The parser is coupled to or integrated with the audio decoder and can parse the metadata container.

[0026] In a typical embodiment, the method includes receiving an encoded audio bitstream that is segmented into one or more frames. The audio data is extracted from the encoded audio bitstream along with a metadata container. The metadata container includes a header, followed by one or more metadata payloads, and followed by protection data. Finally, the integrity of the container and the one or more metadata payloads is verified through the use of the protection data. The one or more metadata payloads may include a program loudness payload that includes data indicating the measured loudness of an audio program associated with the audio data.

[0027] A payload of program loudness metadata embedded in an audio bitstream according to a typical embodiment of the present invention, referred to as loudness processing state metadata (LPSM), may be authenticated and validated so that, for example, a loudness regulatory entity can verify (and thereby ensure compliance with applicable regulations) whether the loudness of a particular program is already within a specified range and that the corresponding audio data itself has not been modified. To verify this, instead of recalculating the loudness, the loudness value contained in the data block containing the loudness processing state metadata may be read. In response to the LPSM, a regulator can determine (as indicated by the LPSM) that the corresponding audio content complies with loudness legislation and / or regulatory requirements (e.g., the Commercial Advertisement Loudness Mitigation Act, also known as the "CALM Act") without having to calculate the loudness of the audio content.

[0028] Loudness measurements required for compliance with certain loudness legislation and / or regulatory requirements (e.g., rules issued under the CALM Act) are based on integrated program loudness. Integrated program loudness requires that loudness measurements, either at dialogue level or full mix level, be performed on the entire audio program. Thus, in order to perform program loudness measurements (e.g., at various stages in the broadcast chain) to verify compliance with typical legal requirements, it is essential that the measurements are made with knowledge of which audio data (and metadata) constitute the entire audio program, which typically requires knowledge of the beginning and end positions of the program (e.g., during the processing of bitstreams showing the sequence of audio programs).

[0029] According to a typical embodiment of the present invention, the encoded audio bitstream represents at least one audio program (e.g., a sequence of audio programs), and the program boundary metadata and LPSM contained in the bitstream enable the reset of the program loudness measurement at the end of the program, thus providing an automated method for measuring the integrated program loudness. The typical embodiment of the present invention includes the program boundary metadata in the encoded audio bitstream in an efficient manner, which allows for accurate and robust determination of at least one boundary between consecutive audio programs represented by the bitstream. The typical embodiment allows for accurate and robust determination of the program boundary, in the sense that even when bitstreams representing different programs are spliced ​​together (to generate the bitstream of the present invention) in a manner that terminates one or both of the spliced ​​bitstreams (thus discarding the program boundary metadata that was contained in at least one of the bitstreams before splicing).

[0030] In a typical embodiment, the program boundary metadata in a bitstream frame of the present invention is a program boundary flag indicating the frame count. Typically, this flag indicates the number of frames between the current frame (the frame containing the flag) and the program boundary (the beginning or end of the current audio program). In some preferred embodiments, the program boundary flag is inserted in a symmetric and efficient manner at the beginning and end of each bitstream segment representing a single program (i.e., in frames occurring within a predetermined number of frames after the beginning of the segment and in frames occurring within a predetermined number of frames before the end of the segment). Thus, when two such bitstreams are concatenated (to represent a sequence of two programs), the program boundary metadata can be present on both sides (e.g., symmetrically) of the boundary between the two programs.

[0031] To limit the data rate increase resulting from including program boundary metadata in an encoded audio bitstream (which may represent a single audio program or a sequence of audio programs), in a typical embodiment, program boundary flags are inserted only in a subset of frames in the bitstream. Typically, the boundary flag insertion rate is a non-increasing function of the increasing distance of each frame of the bitstream (in which the flag is inserted) from the program boundary closest to that frame. Here, “boundary flag insertion rate” represents the average ratio of the number of frames (indicating a program) that contain the program boundary flag to the number of frames (indicating a program) that do not contain the program boundary flag, where the average is a moving average over a number (e.g., a relatively small) of consecutive frames in the encoded audio bitstream. In some embodiments of a certain class, the boundary flag insertion rate is a logarithmically decreasing function of the increasing distance (to each flag insertion location) from the nearest program boundary, and for each flag-containing frame containing one of the flags, the size of the flag in the flag-containing frame is greater than or equal to the size of each flag in a frame located closer to the nearest program boundary than the flag-containing frame (i.e., the size of the program boundary flag in each flag-containing frame is a non-decreasing function of the increasing distance of the flag-containing frame from the nearest program boundary).

[0032] Another aspect of the present invention is an audio processing unit (APU) configured to perform any embodiment of the method of the present invention. In another class of embodiments, the present invention is an APU including a buffer memory (buffer) that stores (for example, in a non-temporary manner) at least one frame of an encoded audio bitstream produced by any embodiment of the method of the present invention. Examples of APUs include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, preprocessing systems (preprocessors), postprocessing systems (postprocessors), audio bitstream processing systems, and combinations of such elements.

[0033] In embodiments of another class, the present invention is an audio processing unit (APU) configured to generate an encoded audio bitstream including audio data segments and metadata segments. The audio data segments represent audio data, and at least some segments of the metadata segments also include loudness processing state metadata (LPSM), and optionally program boundary metadata. Typically, at least one such metadata segment in a frame of bitstream includes at least one segment of LPSM indicating whether a first type of loudness processing has been performed on the audio data of that frame (i.e., the audio data in at least one audio data segment of that frame), and at least one other segment of LPSM indicating the loudness of at least some of the audio data of that frame (e.g., dialogue loudness of at least some of the data showing dialogue in the audio data of that frame). In some embodiments of this class, the APU is an encoder configured to encode input audio to produce encoded audio, and the audio data segments include the encoded audio. In a typical embodiment of this class, each metadata segment has a preferred format as described herein.

[0034] In some embodiments, each metadata segment of an encoded bitstream (in some embodiments, an AC-3 bitstream or an E-AC-3 bitstream) containing an LPSM (e.g., LPSM and program boundary metadata) is included in the extra bits of the skip field segment of the frame of that bitstream (e.g., an extra bit segment W of the type shown in Figure 4 or Figure 7). In other embodiments, each metadata segment of an encoded bitstream (in some embodiments, an AC-3 bitstream or an E-AC-3 bitstream) containing an LPSM (e.g., LPSM and program boundary metadata) is included as additional bitstream information in the "addbsi" field of the bitstream information ("BSI") segment of the frame of that bitstream, or in an auxiliary data field at the end of the frame of that bitstream (e.g., an AUX segment of the type shown in Figure 4 or Figure 7). Each metadata segment containing an LPSM may have the format described in this paper with reference to Tables 1 and 2 below (i.e., including the core elements or variations thereof as described in Table 1, followed by a payload ID (identifying the metadata as an LPSM) and a payload size value, followed by the payload (LPSM data having the format shown in Table 2 or the format shown in the variations to Table 2 described in this paper)). In some embodiments, a frame may contain one or more metadata segments, and if a frame contains two metadata segments, one may be in the addbsi field of the frame and the other in the AUX field of the frame.

[0035] In some embodiments of a certain class, the present invention is a method comprising the step of encoding audio data to produce an AC-3 or E-AC-3 encoded audio stream. The step includes including LPSM and program boundary metadata, and optionally other metadata about the audio program to which the frame belongs, in a metadata segment (of at least one frame of the bitstream). In some embodiments, each such metadata segment is included in the addbsi field of the frame or in the auxiliary data field of the frame. In other embodiments, each such metadata is included in the extra bit segment of the frame. In some embodiments, each metadata segment containing LPSM and program boundary metadata includes a core header (and optionally additional core elements as well) and an LPSM payload (or container) segment having the following format after the core header (or after the core header and other core elements).

[0036] Header. Typically contains at least one identifying value (for example, LPSM format version, length, period, count, and substream association value, as listed in Table 2 of this paper). Following the header, LPSM and program boundary metadata. The program boundary metadata may include the program boundary frame count, a sign value (e.g., "offset_exist" value) indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value, and (if applicable) the offset value.

[0037] LPSM may include the following: At least one dialogue indicator value indicating whether the corresponding audio data contains dialogue or not (for example, which channels of the corresponding audio data contain dialogue). The dialogue indicator value(s) may indicate whether dialogue exists in any combination or all of the channels of the corresponding audio data; At least one loudness compliance value indicating whether the corresponding audio data conforms to the set of loudness regulations; At least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data; and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness).

[0038] In other embodiments, the bitstream to be encoded is a bitstream that is not an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing an LPSM (and optionally program boundary metadata) is contained within a segment (or field or slot) of the bitstream reserved for storing additional data. Each metadata segment containing an LPSM may have the same or identical format as described in this paper with reference to Tables 1 and 2 below (i.e., containing the same or identical core elements as described in Table 1, followed by payload ID and payload size values ​​(identifying the metadata as an LPSM), followed by the payload (LPSM data having the same or identical format as shown in Table 2 or a variation thereof described in this paper)).

[0039] In some embodiments, the encoded bitstream includes a sequence of frames, each frame including a bitstream information ("BSI") segment containing an "addbsi" field (sometimes referred to as a segment or slot) and auxiliary data fields or slots (for example, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream). The bitstream includes audio data segments (for example, the AB0-AB5 segments of the frame shown in Figure 4) and metadata segments, where the audio data segments represent audio data, and at least some segments of the metadata segments include loudness processing state metadata (LPSM) and optionally program boundary metadata. The LPSM exists in the bitstream in the following format. Each metadata segment containing an LPSM is contained in the "addbsi" field of the BSI segment of the frame in the bitstream, or in the auxiliary data fields of the frame in the bitstream, or in the extra bit segments of the frame in the bitstream. Each metadata segment containing an LPSM includes an LPSM payload (or container) segment having the following format.

[0040] Header. Typically includes at least one identifying value, such as the LPSM format version, length, period, count, and substream association values, as shown in Table 2 below; Following the header, the LPSM and optionally the program boundary metadata are also included. The program boundary metadata may include the program boundary frame count, a sign value (e.g., "offset_exist" value) indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value, and (if applicable) the offset value.

[0041] LPSM may include the following: At least one dialogue indicator (e.g., the parameter "Dialogue Channel" in Table 2) indicating whether the corresponding audio data contains dialogue or not (e.g., which channels of the corresponding audio data contain dialogue). The dialogue indicator(s) may indicate whether dialogue exists in any combination or all of the channels of the corresponding audio data; At least one loudness regulation compliance value (for example, the parameter "Loudness Regulation Type" in Table 2) indicating whether the corresponding audio data conforms to the set of loudness regulations specified; At least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data (for example, one or more of the parameters “Dialogue-Gated Loudness Correction Flag” and “Loudness Correction Type” in Table 2); and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness) (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU3341) Short-Time 3s Loudness," and "True Peak" in Table 2).

[0042] In any embodiment of the present invention that considers, uses, or generates at least one loudness value representing corresponding audio data, the loudness value(s) may represent at least one loudness measurement characteristic used to process the loudness and / or dynamic range of the audio data.

[0043] In some implementations, each metadata segment in the "addbsi" field, auxiliary data field, or extra bit segment of the bitstream frame has the following format:

[0044] Core header (typically a synchronization word that identifies the start of a metadata segment, followed by identification values, such as the core element version, length and period, extended element count and substream association values ​​shown in Table 1 below; and After the core header, at least one protective value useful for decoding, authenticating, or validating at least one of the loudness processing state metadata or the corresponding audio data (e.g., HMAC digest and audio fingerprint value, where the HMAC digest may be a 256-bit HMAC digest calculated (using the SHA-2 algorithm) for the audio data of the entire frame, core elements and all expanded elements, as shown in Table 1); and If the metadata segment contains an LPSM after the core header, it includes the LPSM payload identification information ("ID") and the LPSM payload size value, which identify the subsequent metadata as an LPSM payload and indicate the size of the LPSM payload. The LPSM payload segment (preferably in the format described above) is followed by the LPSM payload ID and the LPSM payload size value.

[0045] In some embodiments of the type described above, each metadata segment in the frame's auxiliary data field (or "addbsi" field or extra bit segment) has a three-level structure: High-level structure. This includes a flag indicating whether an auxiliary data (or addbsi) field contains metadata, at least one ID value indicating what type(s) of metadata exists, and typically a value indicating how many bits of metadata (for example, each type) exist (if metadata exists). One type of metadata that can exist is LPSM, another type of metadata that can exist is program boundary metadata, and yet another type of metadata that can exist is media research metadata; Intermediate-level structure. This contains core elements for each identified type of metadata (for example, the core header, protection value and payload ID and payload size values ​​for each identified type of metadata, as mentioned above); and Low-level structure. This includes each payload for a given core element (for example, an LPSM payload if the core element identifies that an LPSM payload exists, and / or another type of metadata payload if the core element identifies that a metadata payload exists).

[0046] Data values ​​in such a three-level structure can be nested. For example, protection values ​​(one or more) for the LPSM payload and / or another metadata payload identified by the core element may be included after each payload identified by the core element (and thus after the core header of the core element). In one example, the core header may identify the LPSM payload and another metadata payload, the payload ID and payload size values ​​for the first payload (e.g., the LPSM payload) may follow the core header, the first payload itself may follow the ID and size values, the payload ID and payload size values ​​for the second payload may follow the first payload, the second payload itself may follow these ID and size values, and protection values ​​(one or more) for one or both of the payloads (or for the core element values ​​and one or both of the payloads) may follow the last payload.

[0047] In some embodiments, the core element of a metadata segment in an auxiliary field of a frame (or "addbsi" field or extra bit segment) includes a core header (typically containing an identification value, e.g., a core element version), followed by: a value indicating whether fingerprint data is included for the metadata of the metadata segment; a value indicating whether external data exists (related to the audio data corresponding to the metadata of the metadata segment); payload ID and payload size values ​​for each type of metadata identified by the core element (e.g., LPSM and / or non-LPSM type metadata); and a protection value for at least one type of metadata identified by the core element. The metadata payload(s) of the metadata segment follow the core header and are (occasionally) nested within the core element's value.

[0048] In another preferred format, the encoded bitstream is a Dolby E bitstream, and each metadata segment, including LPSM (and optionally program boundary metadata as well), is contained within the first N sample positions of the Dolby E protected bandwidth section.

[0049] In embodiments of another class, the present invention is an APU (e.g., a decoder) coupled and configured to receive an encoded audio bitstream including an audio data segment and a metadata segment, where the audio data segment represents audio data, and at least several metadata segments of the metadata segment include loudness processing state metadata (LPSM), and optionally program boundary metadata. The APU is also coupled and configured to extract the LPSM from the bitstream in response to the audio data to generate decoded audio data, and to perform at least one adaptive loudness processing operation on the audio data using the LPSM. Some embodiments of this class also include a post-processor coupled to the APU, which is coupled and configured to perform at least one adaptive loudness processing operation on the audio data using the LPSM.

[0050] In embodiments of another class, the present invention is an audio processing unit (APU) comprising a buffer memory (buffer) and a processing subsystem coupled to the buffer. The APU is coupled to receive an encoded audio bitstream comprising audio data segments and metadata segments. The audio data segments represent audio data, and at least some of each metadata segment of the metadata segments also include loudness processing state metadata (LPSM), and optionally program boundary metadata. The buffer stores at least one frame of the encoded audio bitstream (e.g., non-temporarily), and the processing subsystem is configured to extract the LPSM from the bitstream and to use the LPSM to perform at least one adaptive loudness processing operation on the audio data. In a typical embodiment of this class, the APU is one of an encoder, decoder, and post-processor.

[0051] In some implementations of the method of the present invention, the generated audio bitstream is one of AC-3 encoded bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, and includes loudness processing state metadata and other metadata (e.g., DIALNORM metadata parameter, dynamic range control metadata parameter, and other metadata parameter). In some other implementations of the method, the generated audio bitstream is an encoded bitstream of a different type.

[0052] Aspects of the present invention include a system or apparatus configured (e.g., programmed) to perform any embodiment of the method of the present invention, and a computer-readable medium (e.g., disk) that stores (e.g., in a non-temporary manner) code for implementing any embodiment of the method or steps of the present invention. For example, a system of the present invention may include or be a programmable general-purpose processor, digital signal processor, or microprocessor programmed and / or otherwise configured by software or firmware to perform any of a variety of operations, including embodiments of the method or steps of the present invention, on data. Such a general-purpose processor may be or include a computer system comprising an input device, memory, and processing circuitry programmed (and / or otherwise configured) to perform embodiments of the method (or steps of the present invention) in response to data presented. [Brief explanation of the drawing]

[0053] [Figure 1] This is a block diagram of an embodiment of a system which may be configured to carry out a certain embodiment of the method of the present invention. [Figure 2] This is a block diagram of an encoder, which is an embodiment of the audio processing unit of the present invention. [Figure 3] This is a block diagram of a decoder, which is an embodiment of the audio processing unit of the present invention, and a post-processor, which is another embodiment of the audio processing unit of the present invention, coupled thereto. [Figure 4] This diagram shows an AC-3 frame, including its divided segments. [Figure 5] This diagram shows the synchronization information (SI) segment of an AC-3 frame, including the segments into which it is divided. [Figure 6] This diagram shows the bitstream information (BSI) segment of an AC-3 frame, including the segments into which it is divided. [Figure 7]This diagram shows an E-AC-3 frame, including its divided segments. [Figure 8] This is a diagram of frames of an encoded audio bitstream containing program boundary metadata in a format according to one embodiment of the present invention. [Figure 9] Figure 9 shows other frames of the encoded audio bitstream. Some of these frames include program boundary metadata having a format based on one embodiment of the present invention. [Figure 10] This diagram illustrates two encoded audio bitstreams: one bitstream (IEB) where the program boundary (labeled "boundary") is aligned with the transition between two frames of the bitstream, and another bitstream (TB) where the program boundary (labeled "true boundary") is offset by 512 samples from the transition between two frames of the bitstream. [Figure 11]This is a set of diagrams showing four encoded audio bitstreams. The top bitstream in Figure 11 (labeled "Scenario 1") shows a first audio program (P1) containing program boundary metadata, followed by a second audio program (P2) also containing program boundary metadata; the second bitstream (labeled "Scenario 2") shows a first audio program (P1) containing program boundary metadata, followed by a second audio program (P2) without program boundary metadata; the third bitstream (labeled "Scenario 3") shows a truncated first audio program (P1) containing program boundary metadata, spliced ​​together with the entire second audio program (P2) containing program boundary metadata; and the fourth bitstream (labeled "Scenario 4") shows a truncated first audio program (P1) containing program boundary metadata, followed by a truncated second audio program (P2) containing program boundary metadata, spliced ​​together with a portion of the first audio program. [Modes for carrying out the invention]

[0054] <Notation and Nomenclature> Throughout this disclosure, including the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing prior to the performance of the operation).

[0055] Throughout this disclosure, including the claims, the term “system” is used in a broad sense to mean an apparatus, system, or subsystem. For example, a subsystem that implements a decoder may be called a decoder system, and a system that includes such a subsystem (for example, a system that generates X output signals in response to a plurality of inputs, wherein the subsystem generates M of the inputs and the other XM inputs are received from an external source) may also be called a decoder system.

[0056] Throughout this disclosure, including the claims, the term “processor” is used in a broad sense to mean a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelining operations on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0057] Throughout this disclosure, including the claims, the terms “audio processor” and “audio processing unit” are used interchangeably and in a broad sense to refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools).

[0058] Throughout this disclosure, including the claims, the expression “processing status metadata” (for example, in the expression “loudness processing status metadata”) refers to data separate from the corresponding audio data (the audio content of the audio data stream, including the processing status metadata). The processing status metadata is associated with the audio data and indicates the loudness processing status of the corresponding audio data (for example, which type(s) of processing has already been performed on the audio data), and typically also indicates at least one feature or characteristic of the audio data. The association of the processing status metadata with the audio data is time-synchronous. Thus, the current (most recently received or updated) processing status metadata indicates that the corresponding audio data simultaneously contains the results of the indicated type(s) of audio data processing. In some cases, the processing status metadata may include some or all of the parameters used in and / or derived from the processing of the indicated type in the processing history and / or processing of the indicated type. Furthermore, the processing status metadata may include at least one feature or characteristic of the corresponding audio data that has been computed or extracted from the audio data. Processing status metadata may also include other metadata that is not related to any processing of the corresponding audio data and is not derived from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc., may be added by a particular audio processing unit and passed to other audio processing units.

[0059] Throughout this disclosure, including the claims, the expression “Loudness Processing Status Metadata” (or “LPSM”) means processing status metadata that typically also indicates the loudness processing status of the corresponding audio data (e.g., what type(s) of loudness processing has already been performed on the audio data), and also at least one feature or characteristic (e.g., loudness) of the corresponding audio data. The loudness processing status metadata may include data (e.g., other metadata) that is not loudness processing status metadata (when considered in isolation).

[0060] Throughout this disclosure, including the claims, the term “channel” (or “audio channel”) refers to a monophonic audio signal.

[0061] Throughout this disclosure, including the claims, the expression “audio program” means one or more audio channels and optionally associated metadata (e.g., metadata describing a desired spatial audio presentation and / or LPSM and / or program boundary metadata).

[0062] Throughout this disclosure, including the claims, the expression “program boundary metadata” means metadata of an encoded audio bitstream representing at least one audio program (e.g., two or more audio programs), where the program boundary metadata indicates the position in the bitstream at at least one boundary (beginning and / or end) of the at least one of the audio programs. For example, the program boundary metadata (of an encoded audio bitstream representing an audio program) may include metadata indicating the beginning of the program (e.g., the beginning of the Nth frame of the bitstream or the Mth sample position of the Nth frame of the bitstream) and additional metadata indicating the end of the program (e.g., the beginning of the Jth frame of the bitstream or the Kth sample position of the Jth frame of the bitstream).

[0063] Throughout this disclosure, including the claims, the terms “to combine” or “to be combined” are used to mean direct or indirect connections. Thus, when the first device combines with the second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0064] <Detailed description of embodiments of the invention> According to a typical embodiment of the present invention, a payload of program loudness metadata, referred to as loudness processing state metadata ("LPSM"), and optionally also program boundary metadata, is embedded in one or more reserved fields (or slots) of a metadata segment of an audio bitstream. The audio bitstream also includes audio data in other segments (audio data segments). Typically, at least one segment of each frame of the bitstream includes an LPSM, and at least one other segment of the frame includes the corresponding audio data (i.e., audio data in which the loudness processing state and loudness are indicated by the LPSM). In some embodiments, the data volume of the LPSM may be small enough to be carried without affecting the bitrate allocated to carry the audio data.

[0065] Communicating loudness processing status metadata in an audio data processing chain is particularly useful when two or more audio processing units need to function cascaded with each other throughout the processing chain (or content lifecycle). Without including loudness processing status metadata in the audio bitstream, serious media processing problems such as quality, level, and spatial degradation can occur, for example, when two or more audio codecs are used in the chain and single-ended volume leveling is applied two or more times during the bitstream's journey to the media consumer (or the rendering point of the bitstream's audio content).

[0066] Figure 1 is a block diagram of an exemplary audio processing chain (audio data processing system). Here, one or more elements of the system may be configured according to embodiments of the present invention. The system includes the following elements, coupled together as shown in the figure: a preprocessing unit, an encoder, a signal analysis and metadata correction unit, a transcoder, a decoder, and a preprocessing unit. Variations of the illustrated system may omit one or more elements or include additional audio data processing units.

[0067] In some implementations, the preprocessing unit in Figure 1 is configured to accept a PCM (time-domain) sample containing audio content as input and to output a processed PCM sample. The encoder is configured to accept the PCM sample as input and to output an encoded (e.g., compressed) audio bitstream representing the audio content. The data of the bitstream representing the audio content is sometimes referred to in this paper as "audio data". When the encoder is configured according to a typical embodiment of the present invention, the audio bitstream output from the encoder includes loudness processing state metadata (and other metadata, typically and optionally including program boundary metadata) in addition to the audio data.

[0068] The signal analysis and metadata correction unit in Figure 1 may accept one or more encoded audio bitstreams as input and perform signal analysis (for example, using program boundary metadata in the encoded audio bitstream) to determine whether the processing state metadata in each encoded audio bitstream is correct (e.g., validity check). If the signal analysis and metadata correction unit finds that the included metadata is invalid, the unit replaces the typically incorrect values(s) with the correct values(s) obtained from the signal analysis. Thus, each encoded audio bitstream output from the signal analysis and metadata correction unit may contain corrected (or uncorrected) processing state metadata in addition to the encoded audio data.

[0069] The transcoder in Figure 1 may accept an encoded audio bitstream as input and, in response, output a modified (e.g., encoded in a different way) audio bitstream (e.g., by decoding the input stream and re-encoding the decoded stream in a different encoding format). If the transcoder is configured according to a typical embodiment of the present invention, the audio bitstream output from the transcoder will include loudness processing status metadata (and typically other metadata) in addition to the encoded audio data. This metadata may be included in the bitstream.

[0070] The decoder in Figure 1 accepts an encoded (e.g., compressed) bitstream as input and outputs a decoded stream of PCM audio samples (in response). When the decoder is configured according to a typical embodiment of the present invention, the output of the decoder in typical operation is any of or includes the following: A stream of audio samples and a corresponding stream of loudness processing status metadata (and typically other metadata as well) extracted from the input encoded bitstream; or A corresponding stream of control bits determined from the audio sample stream and loudness processing state metadata (and typically other metadata as well) extracted from the input encoded bitstream; or A stream of audio samples without the corresponding stream of processing state metadata and control bits determined from the processing state metadata. In this last case, the decoder may extract loudness processing state metadata (and / or other metadata) from the input encoded bitstream and perform at least one action on the extracted metadata (e.g., validity check) without outputting the extracted metadata and the control bits determined therefrom.

[0071] By configuring the post-processing unit in Figure 1 based on a typical embodiment of the present invention, the post-processing unit is configured to receive a stream of decoded PCM audio samples and to perform post-processing (e.g., volume leveling of the audio content) on them using loudness processing state metadata (and typically other metadata as well) or control bits (determined by the decoder from the loudness processing state metadata and typically other metadata) received along with the samples. The post-processing unit is also typically configured to render the post-processed audio content for playback by one or more speakers.

[0072] A typical embodiment of the present invention provides an improved audio processing chain in which audio processing units (e.g., encoders, decoders, transcoders, and pre-processing and post-processing units) adapt their respective processing according to the simultaneous state of media data, indicated by loudness processing state metadata received by each audio processing unit.

[0073] Audio data input to any audio processing unit of the system in Figure 1 (e.g., the encoder or transcoder in Figure 1) may include loudness processing status metadata (and optionally other metadata) in addition to the audio data (e.g., encoded audio data). According to one embodiment of the present invention, this metadata may be included in the input audio by other elements of the system in Figure 1 (or other sources not shown in Figure 1). The processing unit receiving the input audio (along with the metadata) performs at least one action (e.g., validity check) on or in response to the metadata (e.g., adaptive processing of the input audio), and is typically also configured to include the metadata, a processed version of the metadata, or control bits determined from the metadata in its output audio.

[0074] A typical embodiment of the audio processing unit (or audio processor) of the present invention is configured to perform adaptive processing on audio data based on the state of the audio data indicated by loudness processing state metadata corresponding to the audio data. In some embodiments, the adaptive processing is loudness processing (or includes loudness processing) (if the metadata indicates that loudness processing or similar processing has not already been performed on the audio data), but not loudness processing (or does not include loudness processing) (if the metadata indicates that such loudness processing or similar processing has already been performed on the audio data). In some embodiments, the adaptive processing is or includes metadata validity verification (e.g., performed in a metadata verification subunit) to ensure that the audio processing unit performs other adaptive processing on the audio data based on the state of the audio data indicated by the loudness processing state metadata. In some embodiments, the validity verification determines the reliability of the loudness processing state metadata associated with the audio data (e.g., included in the bitstream with the audio data). For example, if the metadata is validated as reliable, the results from a previously performed audio processing of a certain type may be reused, and a new execution of the same type of audio processing may be avoided. On the other hand, if it is found that the metadata has been tampered with (or is otherwise unreliable), the audio processing unit may repeat media processing of the type that was supposedly performed previously (as indicated by the unreliable metadata), and / or the audio processing unit may perform other processing on the metadata and / or audio data.An audio processing unit may be configured to signal to other downstream audio processing units in the improved media processing chain that loudness processing state metadata (e.g., present in the media bitstream) is valid, if the unit determines that the processing state metadata is valid (e.g., based on a match between the extracted cryptographic value and the reference cryptographic value).

[0075] Figure 2 is a block diagram of an encoder (100) which is an embodiment of the audio processing unit of the present invention. Any component or element of the encoder 100 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuit) in hardware, software, or a combination of hardware and software. The encoder 100 has a frame buffer 110, a parser 111, a decoder 101, an audio state enable verifier 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a stuffer / formatter stage 107, a metadata generation stage 106, a dialogue loudness measurement subsystem 108, and a frame buffer 109, connected as shown in the figure. Typically, the encoder 100 also includes other processing elements (not shown).

[0076] The encoder 100 (which is a transcoder) is configured to convert an input audio bitstream (which may be, for example, one of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream) into an encoded output audio bitstream (which may be, for example, another of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream). This includes performing adaptive and automated loudness processing using loudness processing state metadata contained in the input bitstream. For example, the encoder 100 may be configured to convert an input Dolby E bitstream (a format typically used in production and broadcast facilities, but not typically used in consumer devices receiving broadcasted audio programs) into an encoded output audio bitstream in the form of AC-3 or E-AC-3 (suitable for broadcasting to consumer devices).

[0077] The system in Figure 2 also includes an encoded audio delivery subsystem 150 (which stores and / or delivers the encoded bitstream output from encoder 100) and a decoder 152. The encoded audio bitstream output from encoder 100 may be stored by subsystem 150 (for example, in the form of a DVD or Blu-ray disc), or transmitted by subsystem 150 (which may implement a transmission link or network), or both stored and transmitted by subsystem 150. Decoder 152 is configured to decode the encoded audio bitstream (generated by encoder 100) received via subsystem 150. This involves extracting loudness processing state metadata (LPSM) from each frame of the bitstream (and optionally extracting program boundary metadata from the bitstream) to generate decoded audio data. Typically, decoder 152 is configured to perform adaptive loudness processing on the decoded audio data using LPSM (and optionally program boundary metadata as well), and / or to transfer the decoded audio data and LPSM to a post-processor configured to perform adaptive loudness processing on the decoded audio data using LPSM (and optionally program boundary metadata as well). Typically, decoder 152 includes a buffer for storing (e.g., non-temporarily) the encoded audio bitstream received from subsystem 150.

[0078] Various implementations of the encoder 100 and decoder 152 are configured to perform various embodiments of the method of the present invention.

[0079] The frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, the buffer 110 stores (for example, non-temporarily) at least one frame of the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream is presented from the buffer 110 to the parser 111.

[0080] The parser 111 is configured to extract loudness processing metadata (LPSM), optionally program boundary metadata (and / or other metadata), from each frame of encoded input audio containing such metadata, and to present at least the LPSM (and optionally program boundary metadata and / or other metadata) to the audio state validity checker 102, loudness processing stages 103, 106 and subsystem 108, extract audio data from the encoded input audio, and present the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode the audio data to produce decoded audio data and to also present the decoded audio data to the loudness processing stage 103, audio stream selection stage 104, subsystem 108 and typically the state validity checker 102.

[0081] The state validity verifier 102 is configured to authenticate and validate the LPSM (and optionally other metadata) presented to it. In some embodiments, the LPSM is (or is contained within) a data block that was included in the input bitstream (for example, according to one embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided from the decoder 101 to the validity verifier 102). In these embodiments, the data block may be digitally signed. This allows downstream audio processing units to authenticate and validate the processing state metadata relatively easily.

[0082] For example, HMAC may be used to generate a digest, and the protection values ​​(one or more) included in the bitstream of the present invention may include the digest. For AC-3 frames, the digest may be generated as follows: 1. After the AC-3 data and LPSM are encoded, the frame data bytes (concatenated frame data #1 and frame data #2) and LPSM data bytes are used as input for the hash function HMAC. Any other data that may be present in the auxiliary data fields is not taken into consideration for calculating this digest. Such other data may be bytes that do not belong to either the AC-3 data or the LPSM data. Protection bits contained in the LPSM may not be taken into consideration for calculating the HMAC digest. 2. After the digest is calculated, it is written to the bitstream in a field reserved for protection bits. 3. The final stage in generating a complete AC-3 frame is the calculation of the CRC check. This is written at the very end of the frame and takes into account all data belonging to this frame, including the LPSM bits.

[0083] Other cryptographic methods, including but not limited to one or more non-HMAC cryptographic methods, may be used for validating the LPSM (e.g., in validator 102) to ensure the secure transmission and reception of the LPSM and / or the underlying audio data. For example, validating (using such cryptographic methods) may be performed in each audio processing unit receiving an embodiment of the audio bitstream of the present invention to determine whether the loudness processing status metadata and corresponding audio data contained in the bitstream have undergone (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified after the execution of such particular loudness processing.

[0084] The status validity checker 102 presents control data to the audio stream selection stage 104, the metadata generator 106, and the dialogue loudness measurement subsystem 108 to indicate the result of the validity check operation. In response to this control data, stage 104 can select (and communicate to encoder 105) one of the following: (For example, when LPSM indicates that the audio data output from decoder 101 has not undergone a particular type of loudness processing, and the control bit from validity checker 102 indicates that LPSM is enabled) the adaptively processed output of loudness processing stage 103; or (For example, when the LPSM indicates that the audio data output from decoder 101 has already undergone a specific type of loudness processing that is to be performed by stage 103, and the control bit from validity checker 102 indicates that the LPSM is valid) the audio data output from decoder 101.

[0085] Stage 103 of the encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from the decoder 101, based on one or more audio data characteristics indicated by the LPSM extracted by the decoder 101. Stage 103 may also be a real-time loudness and dynamic range control processor for the adaptive conversion domain. Stage 103 may receive user input (e.g., user target loudness / dynamic range value or dialnorm value) or other metadata input (e.g., one or more types of third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc.) and / or other information (e.g., from a fingerprinting process) and use such input to process the decoded audio data output from the decoder 101. Stage 103 may perform adaptive loudness processing on decoded audio data (output from decoder 101) that indicates a single audio program (indicated by program boundary metadata extracted by parser 111), and may reset the loudness processing in response to receiving decoded audio data (output from decoder 101) that indicates a different audio program (indicated by program boundary metadata extracted by parser 111).

[0086] The dialogue loudness measurement subsystem 108 may, if the control bit from the enable verifier 102 indicates that the LPSM is disabled, operate to determine the loudness of the segments of the decoded audio (from decoder 101) that represent the dialogue (or other utterances), for example, using the LPSM (and / or other metadata) extracted by decoder 101. If the control bit from the enable verifier 102 indicates that the LPSM is enabled, the operation of the dialogue loudness measurement subsystem 108 may be disabled when the LPSM indicates the previously determined loudness of the dialogue (or other utterances) segments of the decoded audio (from decoder 101). The subsystem 108 may perform a loudness measurement on decoded audio data that represents a single audio program (indicated by program boundary metadata extracted by parser 111), and may reset the measurement in response to receiving decoded audio data that represents a different audio program indicated by such program boundary metadata.

[0087] Useful tools exist for conveniently and easily measuring the level of dialogue in audio content (e.g., Dolby LM100). Some embodiments of the APU of the present invention (e.g., stage 108 of encoder 100) are implemented to include (or perform the functions of) such a tool for measuring the average dialogue loudness of the audio content of an audio bitstream (e.g., the decoded AC-3 bitstream presented from decoder 101 to stage 108 of encoder 100).

[0088] If step 108 is implemented to measure the true average dialogue loudness of audio data, the measurement may include a step of isolating segments of the audio content that primarily contain speech. The audio segments that primarily contain speech are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, this algorithm may be a standard K-weighted loudness measure (according to the international standard ITU-R BS.1770). Alternatively, other loudness measures (e.g., those based on psychoacoustic models of loudness) may be used.

[0089] While isolating speech segments is not essential for measuring the average dialogue loudness of audio data, it improves the accuracy of the metric and typically yields more satisfactory results from the listener's perspective. Since not all audio content contains dialogue (utterances), a loudness metric for the entire audio content can provide a sufficient approximation of the dialogue level of that audio if utterances were present.

[0090] The metadata generator 106 generates (and / or passes to) metadata to be included by stage 107 in the encoded bitstream output from encoder 100. The metadata generator 106 may pass to stage 107 the LPSM (and optionally program boundary metadata and / or other metadata) extracted by encoder 101 and / or parser 111 (for example, if the control bits from validity checker 102 indicate that the LPSM and / or other metadata are valid), or it may generate a new LPSM (and optionally program boundary metadata and / or other metadata) and present the new metadata to stage 107 (for example, if the control bits from validity checker 102 indicate that the LPSM and / or other metadata extracted by decoder 101 are invalid). Alternatively, it may present to stage 107 a combination of the metadata extracted by decoder 101 and / or parser 111 and the newly generated metadata. The metadata generator 106 may include in the LPSM presented to stage 107 the loudness data generated by subsystem 108 and at least one value indicating the type of loudness processing performed by subsystem 108, in the encoded bitstream output from encoder 100.

[0091] The metadata generator 106 may generate useful protection bits (which may consist of, or include, a hash-based message authentication code or "HMAC") for decoding, authenticating, or validating at least one of the LPSM (and optionally other metadata) and / or the underlying audio data to be included in the encoded bitstream.

[0092] In typical operation, the dialogue loudness measurement subsystem 108 processes the audio data output from the decoder 101 and, in response, generates loudness values ​​(e.g., gated and ungated dialogue loudness values) and dynamic range values. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) to be included (by the packer / formatter 107) in the encoded bitstream output from the encoder 100.

[0093] Additionally, optionally, or alternatively, subsystems 106 and / or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicating at least one characteristic of the audio data for inclusion in the encoded bitstream output from stage 107.

[0094] The encoder 105 encodes the audio data output from the selection stage 104 (for example, by performing compression on it) and presents the encoded audio to the stage 107 for inclusion in the encoded bitstream output from the stage 107.

[0095] Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including LPSM) from generator 106 to produce an encoded bitstream output from stage 107. Preferably, the encoded bitstream has a format specified by a preferred embodiment of the present invention.

[0096] The frame buffer 109 is a buffer memory that stores (for example, non-temporarily) at least one frame of the encoded audio bitstream output from stage 107. The sequence of those frames of the encoded audio bitstream is then presented from buffer 109 to the delivery system 150 as output from encoder 100.

[0097] The LPSM generated by the metadata generator 106 and included in the encoded bitstream by stage 107 indicates the loudness processing status of the corresponding audio data (e.g., what type(s) of loudness processing was performed on the audio data) and the loudness of the corresponding audio data (e.g., measured dialogue loudness, gated and / or ungated loudness, and / or dynamic range).

[0098] In this paper, “gating” in level measurements performed on loudness and / or audio data refers to a specific level or loudness threshold above which any calculated value(s) exceeding the threshold are included in the final measurement (for example, ignoring short-term loudness values ​​below -60 dBFS in the final measured value). Gating for absolute values ​​refers to a fixed level or loudness, while gating for relative values ​​refers to a value that depends on the current “ungated” measurement.

[0099] In some implementations of encoder 100, the encoded bitstream buffered in memory 109 (and output to delivery system 150) is an AC-3 bitstream or E-AC-3 bitstream, and includes audio data segments (e.g., AB0-AB5 segments of the frame shown in Figure 4) and metadata segments. Here, the audio data segments represent audio data, and each of at least some segments of the metadata segments contains loudness processing state metadata (LPSM). Stage 107 inserts the LPSM (and optionally program boundary metadata as well) into the bitstream in the following format: Each metadata segment containing the LPSM (and optionally program boundary metadata as well) is included in the extra bit segment of the bitstream (e.g., the extra bit segment "W" shown in Figure 4 or Figure 7) or in the "addbsi" field of the bitstream information ("BSI") segment of the bitstream frame of the bitstream or in an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in Figure 4 or Figure 7). Each bitstream frame may contain one or two metadata segments, each containing an LPSM. If a frame contains two metadata segments, one may be in the frame's addbsi field and the other in the frame's AUX field. In some embodiments, each metadata segment containing an LPSM includes an LPSM payload (or container) segment having the following format: The header (typically containing a synchronization word that identifies the beginning of the LPSM payload, followed by at least one identifier value, such as the LPSM format / version, length, period, count, and substream association value shown in Table 2 below), After the header, At least one dialogue indicator value (for example, the parameter "Dialogue Channel" in Table 2) that indicates whether the corresponding audio data indicates a dialogue or not (for example, which channel of the corresponding audio data indicates a dialogue); At least one loudness regulation compliance value (for example, the parameter "Loudness Regulation Type" in Table 2) indicating whether the corresponding audio data conforms to the set of loudness regulations indicated; At least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data (for example, one or more of the parameters “Dialogue-Gated Loudness Correction Flag” and “Loudness Correction Type” in Table 2); and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness) (e.g., one or more of the parameters "ITU relative gated loudness", "ITU speech gated loudness", "ITU (EBU3341) short-time 3s loudness", and "true peak").

[0100] In some embodiments, each metadata segment containing LPSM and program boundary metadata includes a core header (and optionally additional core elements), followed by the core header (or the core header and other core elements), and including an LPSM payload (or container) segment having the following format: Header. Typically contains at least one identifying value (for example, LPSM format version, length, period, count, and substream association value, as shown in Table 2 of this paper); Following the header, LPSM and program boundary metadata. The program boundary metadata may include the program boundary frame count, a sign value (e.g., "offset_exist" value) indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value, and (if applicable) the offset value.

[0101] In some implementations, each metadata segment inserted by stage 107 into the "addbsi" field or auxiliary data field of the extra bit segment or bitstream frame has the following format: Core header (typically a synchronization word that identifies the start of a metadata segment, followed by identification values, such as the core element version, length and period, extended element count and substream association values ​​shown in Table 1 below); and After the core header, at least one protection value useful for decoding, authenticating, or validating at least one of the loudness processing status metadata or the corresponding audio data (e.g., the HMAC digest and audio fingerprint values ​​in Table 1); and If the metadata segment contains an LPSM after the core header, it includes the LPSM payload identification information ("ID") and the LPSM payload size value, which identify the subsequent metadata as an LPSM payload and indicate the size of the LPSM payload.

[0102] The LPSM payload (or container) segment (preferably in the format specified above) is followed by the LPSM payload ID and LPSM payload size values.

[0103] In some embodiments, each metadata segment in the frame's auxiliary data field (or "addbsi" field) has a three-level structure: High-level structure. This includes a flag indicating whether an auxiliary data (or addbsi) field contains metadata, at least one ID value indicating what type(s) of metadata exists, and typically a value indicating how many bits of metadata (e.g., each type) exist (if metadata exists). One type of metadata that can exist is LPSM, another type of metadata that can exist is program boundary metadata, and another type of metadata that can exist is media research metadata (e.g., Nielsen Media Research metadata); Intermediate-level structure. This contains core elements for each identified type of metadata (for example, the core header, protection value, and LPSM payload ID and LPSM payload size values ​​for each identified type of metadata as described above); and Low-level structure. This includes each payload for a given core element (for example, an LPSM payload if the core element identifies that an LPSM payload exists, and / or another type of metadata payload if the core element identifies that a metadata payload exists).

[0104] Data values ​​in such a three-level structure can be nested. For example, protection values ​​(one or more) for the LPSM payload and / or another metadata payload identified by the core element may be included after each payload identified by the core element (and thus after the core header of the core element). In one example, the core header may identify the LPSM payload and another metadata payload, the payload ID and payload size values ​​for the first payload (e.g., the LPSM payload) may follow the core header, the first payload itself may follow the ID and size values, the payload ID and payload size values ​​for the second payload may follow the first payload, the second payload itself may follow these ID and size values, and protection bits (one or more) for both payloads (or for the core element values ​​and both payloads) may follow the last payload.

[0105] In some embodiments, when decoder 101 receives an audio bitstream generated according to a certain embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a data block determined from the bitstream. The block also includes loudness processing state metadata (LPSM) and optionally program boundary metadata. Validity verifier 102 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if validity verifier 102 finds the LPSM valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, validity verifier 102 may disable the processor 103's operation on the corresponding audio data and allow the audio data to pass through the selection stage 104 (unmodified). Additionally, optionally, or alternatively, other types of cryptographic techniques may be used instead of the cryptographic hash-based method.

[0106] The encoder 100 in Figure 2 may determine (in response to the LPSM extracted by the decoder 101, and optionally also to program boundary metadata) that a post / preprocessing unit has performed a certain type of loudness processing on the audio data to be encoded (in elements 105, 106, and 107), and thus may generate loudness processing state metadata (in generator 106) that includes specific parameters used in and / or derived from the previously performed loudness processing. In some implementations, the encoder 100 may generate (and include in the encoded bitstream output therefrom) processing state metadata indicating the processing history of the audio content, as long as the encoder is aware of the type of processing performed on the audio content.

[0107] Figure 3 is a block diagram of a decoder (200) and a post-processor (300) coupled thereto, which is an embodiment of the audio processing unit of the present invention. The post-processor (300) is also an embodiment of the audio processing unit of the present invention. Any component or element of the decoder 200 and the post-processor 300 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuit) in hardware, software, or a combination of hardware and software. The decoder 200 has a frame buffer 201, a parser 205, an audio decoder 202, an audio state validity check stage (validity checker) 203, and a control bit generation stage 204, connected as shown in the figure. Typically, the decoder 200 also includes other processing elements (not shown).

[0108] The frame buffer 201 (buffer memory) stores (for example, non-temporarily) at least one frame of the encoded audio bitstream received by the decoder 200. The sequence of frames of the encoded audio bitstream is presented from buffer 201 to parser 205.

[0109] The parser 205 is configured to extract loudness processing metadata (LPSM), optionally program boundary metadata and other metadata, from each frame of the encoded input audio, present at least the LPSM (and program boundary metadata, if extracted) to the audio state validator 203 and stage 204, present the LPSM (and optionally program boundary metadata) as output (for example, to the post-processor 300), extract audio data from the encoded input audio, and present the extracted audio data to the decoder 202.

[0110] The encoded audio bitstream input to the decoder 200 may be one of the following: AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream.

[0111] The system in Figure 3 also includes a post-processor 300. The post-processor 300 has a frame buffer 301 and other processing elements (not shown) which include at least one processing element coupled to the buffer 301. The frame buffer 301 stores (e.g., non-temporarily) at least one frame of the decoded audio bitstream received by the post-processor 300 from the decoder 200. The processing elements of the post-processor 300 are coupled and configured to receive a sequence of frames of the decoded audio bitstream output from the buffer 301 and process it adaptively using metadata (including LPSMs) output from the decoder 202 and / or control bits output from stage 204 of the decoder 200. Typically, the post-processor 300 is configured to perform adaptive loudness processing on the decoded audio data using LPSM values ​​and optionally program boundary metadata as well (e.g., based on loudness processing status and / or one or more audio data characteristics indicated by LPSMs for audio data representing a single audio program).

[0112] Various implementations of the decoder 200 and post-processor 300 are configured to perform various embodiments of the method of the present invention.

[0113] The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the parser 205, generate decoded audio data, and present the decoded audio data as output (for example, to the post-processor 300).

[0114] The state validity verifier 203 is configured to authenticate and validate the LPSM (and optionally other metadata) presented to it. In some embodiments, the LPSM is (or is contained within) a data block included in the input bitstream (for example, according to one embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided to the validity verifier 203 from the parser 205 and / or decoder 202). In these embodiments, the data block may be digitally signed. This allows downstream audio processing units to authenticate and validate the processing state metadata relatively easily.

[0115] Other cryptographic methods, including but not limited to one or more non-HMAC cryptographic methods, may be used for validating the LPSM (e.g., in validating verifier 203) to ensure the secure transmission and reception of the LPSM and / or the underlying audio data. For example, validating (using such cryptographic methods) may be performed in each audio processing unit receiving an embodiment of the audio bitstream of the present invention to determine whether the loudness processing status metadata and corresponding audio data contained in the bitstream have undergone (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified after the execution of such particular loudness processing.

[0116] The state validity checker 203 presents control data that controls the bit generator 204 and / or presents said control data as an output (for example, to the post-processor 300) to indicate the result of the validity check operation. In response to said control data (and optionally other metadata extracted from the input bitstream), stage 204 may generate (and present to the post-processor 300) any of the following: (For example, when LPSM indicates that the audio data output from decoder 202 has undergone a specific type of loudness processing, and a control bit from validity checker 203 indicates that LPSM is valid) A control bit indicating that the decoded audio data output from decoder 202 has undergone that specific type of loudness processing; or (For example, when the LPSM indicates that the audio data output from decoder 202 has not undergone a particular type of loudness processing, or when the LPSM indicates that the audio data output from decoder 202 has undergone a particular type of loudness processing, but the control bit from the validity checker 203 indicates that the LPSM is not valid) A control bit indicating that the decoded audio data output from decoder 203 should undergo the particular type of loudness processing.

[0117] Alternatively, decoder 200 may present the metadata extracted from the input bitstream by decoder 202 and the LPSM (and optionally program boundary metadata) extracted from the input bitstream by parser 205 to post-processor 300, which may use the LPSM (and optionally program boundary metadata) to perform loudness processing on the decoded audio data, perform an LPSM validity check, and if the validity check indicates that the LPSM is valid, then use the LPSM (and optionally program boundary metadata) to perform loudness processing on the decoded audio data.

[0118] In some embodiments, when decoder 200 receives an audio bitstream generated according to a certain embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a data block determined from the bitstream. The block includes loudness processing state metadata (LPSM). Validity verifier 203 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if validity verifier 203 finds the LPSM valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, validity verifier 203 may signal a downstream audio processing unit (e.g., a volume leveling unit or a post-processor 300 which may include a volume leveling unit) to pass the audio data of the bitstream through (unmodified). Additionally, optionally, or alternatively, other types of cryptographic techniques may be used instead of the cryptographic hash-based method.

[0119] In some implementations of the decoder 200, the encoded bitstream received (and buffered in memory 201) is an AC-3 bitstream or an E-AC-3 bitstream, which includes an audio data segment (e.g., the AB0-AB5 segments of the frame shown in Figure 4) and a metadata segment. Here, the audio data segment represents audio data, and at least some segments of the metadata segment also include loudness processing state metadata (LPSM) and optionally program boundary metadata. The decoder stage 202 (and / or parser 205) is configured to extract the LPSM (and optionally program boundary metadata) from the bitstream in the following format: Each metadata segment containing the LPSM (and optionally program boundary metadata) is included in the extra bit segment of the frame of the bitstream or in the "addbsi" field of the bitstream information ("BSI") segment of the frame of the bitstream, or in the auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in Figure 4). Each bitstream frame may contain one or two metadata segments, each containing an LPSM. If a frame contains two metadata segments, one may be in the frame's addbsi field and the other in the frame's AUX field. In some embodiments, each metadata segment containing an LPSM includes an LPSM payload (or container) segment having the following format:

[0120] Header (typically containing a synchronization word that identifies the beginning of the LPSM payload, followed by identification values, such as the LPSM format / version, length, period, count, and substream association values ​​shown in Table 2 below); After the header, At least one dialogue indicator value (for example, the parameter "Dialogue Channel" in Table 2) that indicates whether the corresponding audio data indicates a dialogue or not (for example, which channel of the corresponding audio data indicates a dialogue); At least one loudness regulation compliance value (for example, the parameter "Loudness Regulation Type" in Table 2) indicating whether the corresponding audio data conforms to the set of loudness regulations specified; At least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data (for example, one or more of the parameters “Dialogue-Gated Loudness Correction Flag” and “Loudness Correction Type” in Table 2); and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness) (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU3341) Short-Time 3s Loudness," and "True Peak" in Table 2).

[0121] In some embodiments, each metadata segment containing LPSM and program boundary metadata includes a core header (and optionally additional core elements), followed by the core header (or the core header and other core elements), and including an LPSM payload (or container) segment having the following format: Header. Typically contains at least one identifying value (e.g., LPSM format version, length, period, count, and substream association value, as shown in Table 2 below); Following the header, LPSM and program boundary metadata. The program boundary metadata may include the program boundary frame count, a sign value (e.g., "offset_exist" value) indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value, and (if applicable) the offset value.

[0122] In some implementations, the parser 205 (and / or decoder stage 202) is configured to extract each metadata segment from the extra bit segments or "addbsi" field or auxiliary data field of the bitstream frame, having the following format: Core header (typically a synchronization word that identifies the start of a metadata segment, followed by at least one identifying value, such as the core element version, length and period, extended element count and substream association value shown in Table 1 below); and After the core header, at least one protection value useful for decoding, authenticating, or validating at least one of the loudness processing status metadata or the corresponding audio data (e.g., the HMAC digest and audio fingerprint values ​​in Table 1); and If the metadata segment contains an LPSM after the core header, it includes the LPSM payload identification information ("ID") and the LPSM payload size value, which identify the subsequent metadata as an LPSM payload and indicate the size of the LPSM payload.

[0123] The LPSM payload (or container) segment (preferably in the format specified above) is followed by the LPSM payload ID and LPSM payload size values.

[0124] More generally, the encoded audio bitstream generated by a preferred embodiment of the present invention has a structure that provides a mechanism for labeling metadata elements and sub-elements as core (required) or extension (optional elements). This allows scaling the data rate of the bitstream (including metadata) across a number of applications. The core (required) elements of the preferred bitstream syntax should also signal that extension (optional) elements associated with the audio content are present (in-band) and / or at remote locations (out of band).

[0125] Core elements (one or more) are required to be present in every frame of the bitstream. Some sub-elements of the core elements are optional and may be present in any combination. Extension elements are not required to be present in every frame (to limit bitrate overhead). Thus, extension elements may be present in some frames and not in others. Some sub-elements of an extension element are optional and may be present in any combination, but some sub-elements of an extension element may be required (i.e., required if the extension element is present in a frame of the bitstream).

[0126] In one class of embodiments, an encoded audio bitstream is generated (for example, by an audio processing unit embodying the present invention) that includes a sequence of audio data segments and metadata segments. The audio data segments represent audio data, and each of at least some segments of the metadata segments also includes loudness processing state metadata (LPSM) and optionally program boundary metadata, and the audio data segments are time-division multiplexed with the metadata segments. In a preferred embodiment of this class, each of the metadata segments has a preferred format as described herein.

[0127] In one preferred format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing LPSMs is included as additional bitstream information in the "addbsi" field (shown in Figure 6) of the bitstream information ("BSI") segment of the bitstream frame, or in the auxiliary data field of the bitstream frame, or in the extra bit segment of the bitstream frame (for example, by stage 107 of a preferred implementation of encoder 100).

[0128] In the preferred format described above, each frame contains a core element in the frame's addbsi field (or extra bit segment) having the format shown in Table 1 below.

[0129] [Table 1] In this preferred format, each addbsi (or auxiliary data) field or extra bit segment containing an LPSM includes a core header (and optionally additional core elements) and the following LPSM value (parameter) after the core header (or after the core header and other core elements): Payload ID (identifies the metadata as an LPSM). This follows a core element value (such as those specified in Table 1, for example); Payload size (indicates the size of the LPSM payload). This follows the payload ID; LPSM data (followed by payload ID and payload size value). This has the format shown in the following table (Table 2).

[0130] [Table 2-1] [Table 2-2] In another preferred format for the encoded bitstream generated according to the present invention, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments containing LPSM (and optionally program boundary metadata as well) is included in any of the following (e.g., by stage 107 of a preferred implementation of encoder 100): the extra bit segment of the frame of the bitstream; the "addbsi" field of the bitstream information ("BSI") segment of the frame of the bitstream (shown in Figure 6); or the auxiliary data field at the end of the frame of the bitstream (e.g., the AUX field shown in Figure 4). Each frame may contain one or two metadata segments containing LPSM, and if a frame contains two metadata segments, one may reside in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment containing an LPSM has the format defined above, referring to Tables 1 and 2 above (i.e., it includes the core elements specified in Table 1, followed by the Payload ID (which identifies the metadata as an LPSM) and Payload Size value defined above, and then the Payload (LPSM data in the format shown in Table 2)).

[0131] In another preferred format, the encoded bitstream is a Dolby E bitstream, and each metadata segment containing LPSM (and optionally program boundary metadata as well) is the first N sample position of the Dolby E protected bandwidth section. A Dolby E bitstream containing such metadata segments containing LPSM preferably includes a value indicating the LPSM payload length signaled in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition rate preferably remains the same as the associated video frame rate).

[0132] In a preferred format in which the encoded bitstream is an E-AC-3 bitstream, each metadata segment containing LPSM (and optionally program boundary metadata as well) is included as additional bitstream information (for example, by stage 107 of a preferred implementation of encoder 100) in the extra bit segment of the bitstream frame, or in the "addbsi" field of the bitstream information ("BSI") segment. Further aspects of encoding an E-AC-3 bitstream with LPSM in this preferred format are described below.

[0133] 1. During the generation of an E-AC-3 bitstream, while the E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "active," for all generated frames (synchronous frames), the bitstream should include a metadata block (containing LPSMs) carried in the frame's addbsi field (or extra bit segment). The bits required to carry this metadata block should not increase the encoder bitrate (frame length).

[0134] 2. All metadata blocks (including LPSMs) should contain the following information: loudness_correction_type_flag: Here, "1" indicates that the loudness of the corresponding audio data was corrected upstream of the encoder, and "0" indicates that the loudness was corrected by a loudness corrector built into the encoder (for example, the loudness processor 103 of encoder 100 in Figure 2); speech_channel: Indicates which source channel (singular or plural) contains the utterance (within the preceding 0.5 seconds). If no utterance is detected, this will be indicated. speech_loudness: Indicates the combined speech loudness (over the preceding 0.5 seconds) of each corresponding audio channel containing the utterance; ITU_loudness [ITU loudness]: Indicates the integrated ITU BS.1770-3 loudness for each corresponding audio channel; Gain: The composite loudness gain (single or multiple) used for inversion in the decoder (to demonstrate reversibility).

[0135] 3. While the E-AC-3 encoder (inserting LPSM values ​​into the bitstream) is "active" and receiving AC-3 frames with the "trust" flag, the loudness controller in that encoder (e.g., the loudness processor 103 of encoder 100 in Figure 2) should be bypassed. The "trusted" source dialnorm and DRC values ​​should be passed to the E-AC-3 encoder component (e.g., stage 107 of encoder 100) (e.g., by the generator 106 of encoder 100). LPSM block generation continues and loudness_correction_type_flag is set to "1". The loudness controller bypass sequence needs to be synchronized to the beginning of the decoded AC-3 frame in which the "trust" flag appears. The loudness controller bypass sequence should be implemented as follows: The leveler_amount control is decremented from value 9 to value 0 over 10 audio block periods (i.e., 53.3 msec), and the leveler_back_end_meter control is put into bypass mode (this operation should provide a seamless transition). The term "trusted" bypass for the leveler implies that the dialnorm value of the source bitstream is also reused at the encoder output (for example, if the "trusted" source bitstream has a dialnorm value of -30, the encoder output should utilize -30 for the outgoing dialnorm value). While the E-AC-3 encoder (inserting LPSM values ​​into the bitstream) is "active" and receiving AC-3 frames without the "trusted" flag, the loudness controller built into the encoder (e.g., the loudness processor 103 of encoder 100 in Figure 2) should be active. LPSM block generation continues, and loudness_correction_type_flag is set to "0". The loudness controller activation sequence should be synchronized to the beginning of the decoded AC-3 frame where the "trust" flag disappears.The loudness controller activation sequence should be implemented as follows: The leveler_amount control is incremented from value 0 to value 9 over one audio block duration (i.e., 5.3 msec), and the leveler_back_end_meter control is set to "active" mode (this operation should provide a seamless transition and include a back_end_meter integrated reset).

[0136] 5. During encoding, the graphical user interface (GUI) should display the following parameters to the user: "Input audio program [trusted / untrusted]" - the state of this parameter is based on the presence of the "trusted" flag in the input signal; and "Real-time loudness correction: [enable / disabled]" - the state of this parameter is based on whether this loudness controller built into the encoder is active.

[0137] When decoding an AC-3 or E-AC-3 bitstream that has LPSMs contained in the "addbsi" field of the extra bit segment or bitstream information ("BSI") segment of each frame of the bitstream (in the preferred format described above), the decoder should parse the LPSM block data (in the extra bit segment or addbsi field) and pass all extracted LPSM values ​​to the graphical user interface (GUI). The set of extracted LPSM values ​​is refreshed for each frame.

[0138] In another preferred format for the encoded bitstream generated according to the present invention, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing LPSMs is included (for example, by stage 107 of a preferred implementation of encoder 100) in the extra bits segment, or in the Aux segment, or as additional bitstream information in the “addbsi” field (shown in Figure 6) of the bitstream information (“BSI”) segment of the bitstream frame. (This is a variation of the format described above with reference to Tables 1 and 2.) In this format, each addbsi (or Aux or extra bits) field containing LPSMs contains the following LPSM values:

[0139] The core elements are defined in Table 1. These are followed by the payload ID (which identifies the metadata as an LPSM) and the payload size value, followed by the payload (LPSM data). The LPSM data has the following format (similar to the required elements shown in Table 2 above).

[0140] LPSM Payload Version: A 2-bit field indicating the version of the LPSM payload.

[0141] dialchan: A 3-bit field indicating whether the left, right, and / or center channels of the corresponding audio data contain spoken dialogue. The bit assignment of the dialchan field may be as follows: Bit 0, indicating the presence of dialogue in the left channel, is stored in the most significant bit of the dialchan field, and Bit 2, indicating the presence of dialogue in the center channel, is stored in the least significant bit of the dialchan field. Each bit of the dialchan field is set to "1" if the corresponding channel contains dialogue spoken during the preceding 0.5 seconds of the program.

[0142] loudregtyp: A 4-bit field indicating which regulatory standards the program loudness complies with. Setting the "loudregtyp" field to "000" indicates that the LPSM does not indicate compliance with loudness regulations. For example, one value for this field (e.g., 0000) may indicate that compliance with loudness regulations is not indicated, another value for this field (e.g., 0001) may indicate that the program's audio data complies with the ATSC A / 85 standard, and yet another value for this (e.g., 0010) may indicate that the program's audio data complies with the EBU R128 standard. In this example, if this field is set to any value other than "0000", the loudcorrdialgat and loudcorrtyp fields should follow the payload.

[0143] loudcorrdialgat: A 1-bit field indicating whether dialog-gated loudness correction has been applied. If the program's loudness is corrected using dialog gating, the value of the loudcorrdialgat field is set to "1". Otherwise, its value is set to "0".

[0144] loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. The loudcorrtyp field is set to "0" if the program's loudness is corrected using an infinite lookahead (file-based) loudness correction process. The value of this field is set to "1" if the program's loudness is corrected using a combination of real-time loudness measurement and dynamic range control.

[0145] loudrelgate: A 1-bit field indicating whether relative gated loudness data (ITU) exists. If the loudrelgate field is set to "1", the payload should be followed by a 7-bit ituloudrelgate field.

[0146] loudrelgat: A 7-bit field indicating relative gated program loudness (ITU). This field represents the integrated loudness of the audio program, measured according to ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in increments of 0.5 LKFS.

[0147] loudspchgate: A 1-bit field indicating whether gated loudness data (ITU) exists in the utterance. If the loudspchgate field is set to "1", the payload should be followed by a 7-bit loudspchgate field.

[0148] loudspchgat: A 7-bit field indicating the speech-gated program loudness. This field represents the integrated loudness of the corresponding audio program as measured according to formula (2) of ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in increments of 0.5 LKFS.

[0149] loudstrm3se: A 1-bit field indicating whether short-term (3-second) loudness data exists. If this field is set to "1", the payload should be followed by a 7-bit loudstrm3s field.

[0150] loudstrm3s: A 7-bit field indicating the ungated loudness of the preceding 3 seconds of the corresponding audio program, measured according to ITU-R BS.1771-1, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 256 are interpreted as -116 LKFS to +11.5 LKFS in increments of 0.5 LKFS.

[0151] truepke: A 1-bit field indicating whether true peak loudness data exists. If the truepke field is set to "1", an 8-bit truepk field should follow in the payload.

[0152] truepk: An 8-bit field indicating the true peak sample value of the program, measured according to Annex 2 of ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 256 are interpreted as -116 LKFS to +5.5 LKFS in increments of 0.5 LKFS.

[0153] In some embodiments, the core element of a metadata segment in an extra bit segment or auxiliary data (or "addbsi") field of a frame of an AC-3 bitstream or E-AC-3 bitstream includes a core header (typically an identification value, e.g., core element version), followed by: a value indicating whether fingerprint data (or other protection values) are included for the metadata of the metadata segment; a value indicating whether external data exists (related to the audio data corresponding to the metadata of the metadata segment); payload ID and payload size values ​​for each type of metadata identified by the core element (e.g., LPSM and / or non-LPSM type metadata); and protection values ​​for at least one type of metadata identified by the core element. The metadata payload(s) of the metadata segment follow the core header and are (occasionally) nested within the core element's value.

[0154] A typical embodiment of the present invention includes program boundary metadata within an encoded audio bitstream in an efficient manner that allows for accurate and robust determination of at least one boundary between consecutive audio programs represented by the bitstream. The typical embodiment allows for accurate and robust determination of program boundaries, in the sense that it allows for accurate program boundary determination even when bitstreams representing different programs are spliced ​​together (to generate the bitstream of the present invention) in a manner that terminates one or both of the spliced ​​bitstreams (thus discarding the program boundary metadata that was contained in at least one of the bitstreams before splicing).

[0155] In a typical embodiment, the program boundary metadata in a bitstream frame of the present invention is a program boundary flag indicating the frame count. Typically, this flag indicates the number of frames between the current frame (the frame containing the flag) and the program boundary (the beginning or end of the current audio program). In some preferred embodiments, the program boundary flag is inserted in a symmetric and efficient manner at the beginning and end of each bitstream segment representing a single program (i.e., in frames occurring within some predetermined number of frames after the beginning of the segment and in frames occurring within some predetermined number of frames before the end of the segment). Thus, when two such bitstreams are concatenated (thus representing a sequence of two programs), the program boundary metadata can be present on both sides (for example, symmetrically) of the boundary between the two programs.

[0156] Maximum robustness can be achieved by inserting program boundary flags into every frame of the bitstream that indicates a program, but this is typically impractical due to the associated increase in data rate. In a typical embodiment, program boundary flags are inserted into only a subset of frames of the encoded audio bitstream (which may indicate a single audio program or a sequence of audio programs), and the boundary flag insertion rate is a non-increasing function of the increasing distance of each frame of the bitstream (in which the flag is inserted) from the program boundary closest to that frame. Here, “boundary flag insertion rate” represents the average ratio of the number of frames (indicating a program) that contain the program boundary flag to the number of frames (indicating a program) that do not contain the program boundary flag, where the average is a moving average over a number (e.g., a relatively small) of consecutive frames of the encoded audio bitstream.

[0157] Increasing the boundary flag insertion rate (at locations in the bitstream closer to the program boundary) increases the data rate required for bitstream delivery. To compensate for this, it is preferable that the size (number of bits) of each inserted flag decreases as the boundary flag insertion rate increases (so that, where N is an integer, the size of the program boundary flag in the Nth frame of the bitstream is a non-increasing function of the distance (number of frames) between the Nth frame and the nearest program boundary). In some embodiments of a certain class, the boundary flag insertion rate is a logarithmically decreasing function of the increasing distance (to each flag insertion location) from the nearest program boundary, and for each flag-containing frame containing one of the flags, the size of the flag in the flag-containing frame is greater than or equal to the size of each flag in a frame located closer to the nearest program boundary than the flag-containing frame. Typically, the size of each flag is determined by an increasing function of the number of frames from the flag insertion location to the nearest program boundary.

[0158] For example, consider the embodiments shown in Figures 8 and 9. Here, each column identified by the frame number (top row) represents a frame of the encoded audio bitstream. The bitstream represents an audio program with a first program boundary (indicating the start of the program) appearing immediately to the left of the column identified by frame number "17" on the left side of Figure 9, and a second program boundary (indicating the end of the program) appearing immediately to the right of the column identified by frame number "1" on the right side of Figure 8. The program boundary flags included in the frames shown in Figure 8 count down the number of frames between the current frame and the second program boundary. The program boundary flags included in the frames shown in Figure 9 count up the number of frames between the current frame and the first program boundary.

[0159] In the embodiments of Figures 8 and 9, the program boundary flag is set after the start of the audio program indicated by the bitstream, for the first X frames of the encoded bitstream. N In each of the second frames, and the two closest to the end of the program shown by the bitstream (the last X frames of the bitstream) N This is shown only in each of the nth frames. Here, the program contains Y frames, X is an integer less than or equal to Y / 2, and N is a positive integer in the range from 1 to log2X. Thus, (as shown in Figures 8 and 9) the program boundary flag is inserted in the second frame (N=1) of the bitstream (the flag-containing frame closest to the beginning of the program), the fourth frame (N=2), the eighth frame (N=3), etc., and also in the eighth frame from the end of the bitstream, the fourth frame from the end of the bitstream, and the second frame from the end of the bitstream (the flag-containing frame closest to the end of the program). In this example, the program boundary flag is inserted from the 2nd frame from the beginning (or end) of the program. N The program boundary flag in the second frame is log2(2 N+2 ) has binary bits. Thus, the program boundary flag in the second frame from the beginning (or end) of the program (N=1) is log2(2 N+2 ) = log2(2 3 )=3 binary bits, and the flag in the fourth frame (N=2) from the beginning (or end) of the program is log2(2 N+2 ) = log2(2 4 ) = 4 binary bits, and so on.

[0160] In the examples of FIGS. 8 and 9, the format of each program boundary flag is as follows. Each program boundary flag consists of a leading "1" bit, a sequence of "0" bits (no "0" bits or one or more consecutive "0" bits) after the leading "1" bit, and a 2-bit trailing code. For the flags in the last X frames of the bit stream (the frames closest to the end of the program), the trailing code is "11" as shown in FIG. 8. For the flags in the first X frames of the bit stream (the frames closest to the start of the program), the trailing code is "10" as shown in FIG. 9. Thus, to read (decode) each flag, the number of 0s between the leading "1" bit and the trailing code is counted. If the trailing code is identified as "11", the flag indicates that there are (2 Z+1 -1) frames between the current frame (the frame containing that flag) and the end of the program. Here, Z is the number of 0s between the leading "1" bit of this flag and the trailing code. This decoder can be efficiently implemented to ignore the first and last bits of each such flag and determine the reverse of the sequence of the other (intermediate) bits of the flag (e.g., if the sequence of intermediate bits is "0001" and the "1" bit is the last bit of the sequence, the reversed sequence of the intermediate bits becomes "1000" and the "1" bit becomes the first bit of the reversed sequence), and identify the binary value of the reversed sequence of the intermediate bits as the index of the current frame (the frame containing that flag) relative to the end of the program. For example, if the reversed sequence of the intermediate bits is "1000", this reversed sequence has a binary value of 2 4 = 16, and this frame is identified as the 16th frame before the end of the program (as shown in the column describing frame "0" in FIG. 8).

[0161] If the trailing code is identified as "10", the flag indicates that there are (2Z+1 This indicates that there are -1) frames, where Z is the number of zeros between the leading "1" bit and the trailing sign of this flag. This decoder can be efficiently implemented to ignore the first and last bits of each such flag, determine the inverse of the sequence of intermediate bits of the flag (for example, if the sequence of intermediate bits is "0001" and the "1" bit is the last bit of the sequence, then the inverted sequence of intermediate bits would be "1000" and the "1" bit would be the first bit of the inverted sequence), and identify the binary value of the inverted sequence of intermediate bits as the index of the current frame (the frame containing that flag) relative to the beginning of the program. For example, if the inverted sequence of intermediate bits is "1000", then this inverted sequence is the binary value 2 4 It has a value of =16, and this frame is identified as the 16th frame after the start of the program (as shown in the column describing frame "32" in Figure 9).

[0162] In the examples in Figures 8 and 9, the program boundary flag is set for the first X frames of the encoded bitstream after the start of the audio program, as indicated by the bitstream. N In each of the second frames, and the two frames closest to the end of the audio program indicated by the bitstream (the last X frames of the bitstream) N Each of the nth frames simply exists, the program has Y frames, X is an integer less than or equal to Y / 2, and N is a positive integer in the range of 1 to log2X. Including program boundary flags only adds an average bitrate of 1.875 bits / frame to the bitrate required to send the bitstream without flags.

[0163] In a typical implementation of the embodiments shown in Figures 8 and 9, where the bitstream is an AC-3 encoded audio bitstream, each frame contains audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48 kHz, this represents a rate of 32 milliseconds of digital audio or audio at 31.25 frames per second. Thus, in such embodiments, a program boundary flag in a frame that is separated from the program boundary by a certain number of frames ("X" frames) indicates that the boundary appears 32X milliseconds after the end of the flag-containing frame (or 32X milliseconds before the start of the flag-containing frame).

[0164] In a typical implementation of the embodiments shown in Figures 8 and 9, where the bitstream is an E-AC-3 encoded audio bitstream, each frame of the bitstream contains audio content and metadata for 256, 512, 768, or 1536 samples of digital audio, depending on whether the frame contains one, two, three, or six blocks of audio data. For a sampling rate of 48 kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or 189.9, 93.75, 62.5, or 31.25 frames per second of audio, respectively. Thus, in such embodiments, a program boundary flag in a frame that is separated from the program boundary by a certain number of frames ("X" frames) (assuming each frame represents 32 milliseconds of digital audio) indicates that the boundary appears 32X milliseconds after the end of the flag-containing frame (or 32X milliseconds before the start of the flag-containing frame).

[0165] In some embodiments where a program boundary can occur within a frame of an audio bitstream (i.e., not aligned to the beginning or end of a frame), the program boundary metadata contained within the bitstream frame includes a program boundary frame count (i.e., metadata indicating the number of complete frames between the beginning or end of the frame containing the frame count and the program boundary) and an offset value. The offset value indicates the offset (typically a number of samples) between the beginning or end of the frame containing the program boundary and the actual position of the program boundary within that frame.

[0166] An encoded audio bitstream may represent a sequence of program (soundtrack) corresponding to a sequence in a video program, and such audio program boundaries tend to occur at the edges of video frames rather than audio frames. Furthermore, some audio codecs (e.g., E-AC-3 codecs) use audio frame sizes that are not aligned with video frames. Also, in some cases, the initially encoded audio bitstream undergoes transcoding to produce a transcoded bitstream, which has a different frame size than the transcoded bitstream. Therefore, program boundaries (determined by the initially encoded bitstream) are not guaranteed to appear at the frame boundaries of the transcoded bitstream. For example, if the initially encoded bitstream (e.g., bitstream "IEB" in Figure 10) has a frame size of 1536 samples per frame, and the transcoded bitstream (e.g., bitstream "TB" in Figure 10) has a frame size of 1024 samples per frame, the transcoding process may cause the actual program boundary to appear somewhere within the frame of the transcoded bitstream (for example, 512 samples into the frame of the transcoded bitstream, as shown in Figure 10) rather than at the frame boundary of the transcoded bitstream due to the different frame sizes of the different codecs. Embodiments of the present invention in which the program boundary metadata contained in the frames of the encoded audio bitstream includes an offset value in addition to the program boundary frame count are useful in the three cases (and others) described in this paragraph.

[0167] The embodiments described above with reference to Figures 8 and 9 do not include an offset value (e.g., an offset field) in any of the frames of the encoded bitstream. In variations of this embodiment, the offset value is included in each frame of the encoded audio bitstream that contains the program boundary flag (for example, in the frames corresponding to the frames numbered 0, 8, 12, and 14 in Figure 8 and the frames numbered 18, 20, 24, and 32 in Figure 9).

[0168] In some embodiments of a certain class, the data structure (in each frame of an encoded bitstream containing program boundary metadata of the present invention) includes a sign value indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value. For example, the sign value may be the value of a single bit field (referred to hereby as the "offset_exist" field). A value of offset_exist=0 may indicate that the frame does not contain an offset value, while a value of offset_exist=1 may indicate that the frame contains both the program boundary frame count and the offset value.

[0169] In some embodiments, at least one frame of an AC-3 or E-AC-3 encoded audio bitstream includes a metadata segment containing LPSM and program boundary metadata (and optionally other metadata as well) about an audio program determined by the bitstream. Each such metadata segment (which may be contained in the addbsi field or auxiliary data field or extra bit segment of the bitstream) includes a core header (and optionally additional core elements as well) and an LPSM payload (or container) segment having the following format after the core header (or after the core header and other core elements):

[0170] Header (typically including at least one identifying value, such as LPSM format version, length, period, count, and substream association value), Following the header, there is program boundary metadata (which may include the program boundary frame count, a sign value indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value (e.g., an "offset_exist" value), and possibly the offset value) and LPSM. The LPSM may include the following: At least one dialogue indicator value indicating whether the corresponding audio data contains dialogue or not (for example, which channels of the corresponding audio data contain dialogue). The dialogue indicator value(s) may indicate whether dialogue exists in any combination or all of the channels of the corresponding audio data; At least one loudness compliance value indicating whether the corresponding audio data conforms to the set of loudness regulations; At least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data; and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness).

[0171] In some embodiments, the LPSM payload segment includes a sign value (e.g., an "offset_exist" value) indicating whether the frame contains only the program boundary frame count or both the program boundary frame count and the offset value. For example, in one such embodiment, when such a sign value (e.g., offset_exist=1) indicates that the frame contains both the program boundary frame count and the offset value, the LPSM payload segment may be an 11-bit unsigned integer (i.e., having a value from 0 to 2048) and include an offset value indicating the number of additional audio samples between the signaled frame boundary (the boundary of the frame containing the program boundary) and the actual program boundary. If the program boundary frame count indicates the number of frames (at the current frame rate) up to the program boundary-containing frame, then the precise position (in samples) of the program boundary (relative to the beginning or end of the frame containing the LPSM payload segment) is: S=(frame_counter*frame size)+offset It is calculated as follows: where S is the number of samples from the beginning or end of the frame containing the LPSM payload segment to the program boundary. "frame_counter" is the frame count indicated by the program boundary frame count, "frame size" is the number of samples per frame, and "offset" is the number of samples indicated by the offset value.

[0172] In some embodiments where the program boundary flag insertion rate increases near the actual program boundary, a rule is implemented that if a frame is less than or equal to a certain number ("Y") frames from the frame containing the program boundary, that frame will never contain an offset value. Typically, Y=32. For an E-AC-3 encoder that implements this rule (with Y=32), the encoder will never insert an offset value in the last second of the audio program. In this case, the receiving device is responsible for maintaining a timer and thereby performing its own offset calculation (in response to program boundary metadata containing the offset value in frames of the encoded bitstream that are more than Y frames away from the program boundary-containing frame).

[0173] For programs where the audio program is known to be "frame-aligned" with the video frames of the corresponding video program (for example, a typical contributing feed with Dolby E encoded audio), including the offset value in the encoded bitstream representing the audio program is superfluous. Therefore, the offset value is typically not included in such encoded bitstreams.

[0174] Referring to Figure 11, we now consider the case where the encoded audio bitstreams are spliced ​​together to produce an embodiment of the audio bitstream of the present invention.

[0175] The top bitstream in Figure 11 (labeled "Scenario 1") shows the entirety of a first audio program (P1) containing program boundary metadata (program boundary flag F), followed by the entirety of a second audio program (P2), also containing program boundary metadata (program boundary flag F). The program boundary flags at the end of the first program (some of which are shown in Figure 11) are identical or similar to those described with reference to Figure 8 and determine the location of the boundary between the two programs (i.e., the boundary at the beginning of the second program). The program boundary flags at the beginning of the second program (some of which are shown in Figure 11) are identical or similar to those described with reference to Figure 9 and also determine the location of the boundary. In a typical embodiment, an encoder or decoder implements a timer (calibrated by a flag in the first program) that counts down to the program boundary, and the same timer (calibrated by a flag in the second program) that counts up from the same program boundary. As shown by the boundary timer graph in Scenario 1 of Figure 11, the countdown of such a timer (calibrated by a flag in the first program) reaches 0 at the boundary, and the countup of the timer (calibrated by a flag in the second program) refers to the same position at the boundary.

[0176] The second bitstream from the top in Figure 11 (labeled "Scenario 2") shows the entirety of a first audio program (P1) containing program boundary metadata (program boundary flag F), followed by the entirety of a second audio program (P2) that does not contain program boundary metadata. The program boundary flag at the end of the first program (some of which are shown in Figure 11) is identical or similar to those described with reference to Figure 8, and, as in Scenario 1, determines the location of the boundary between the two programs (i.e., the boundary at the beginning of the second program). In a typical embodiment, the encoder or decoder implements a timer (calibrated by a flag in the first program) that counts down to the program boundary, and the same timer continues to count up from that program boundary (without further calibration) (as shown by the boundary timer graph in Scenario 2 in Figure 11).

[0177] The third bitstream from the top in Figure 11 (labeled "Scenario 3") shows a truncated first audio program (P1) containing program boundary metadata (program boundary flag F), spliced ​​with the entire second audio program (P2), which also contains program boundary metadata (program boundary flag F). The splice removes the last "N" frames of the first program. The program boundary flags at the beginning of the second program (some of which are shown in Figure 11) are identical or similar to those described with reference to Figure 9, and determine the location of the boundary (splice) between the truncated first program and the complete second program. In a typical embodiment, an encoder or decoder implements a timer (calibrated by a flag in the first program) that counts down to the end of the untruncated first program, and the same timer (calibrated by a flag in the second program) counts up from the beginning of the second program. The beginning of the second program is the program boundary in Scenario 3. As shown by the boundary timer graph in Scenario 3 of Figure 11, the countdown of such a timer (calibrated by the program boundary metadata in the first program) is reset (in response to the program boundary metadata in the second program) before it reaches 0 (in response to the program boundary metadata in the first program). Thus, the termination of the first program (by splicing) prevents the timer from identifying the program boundary between the terminated first program and the beginning of the second program in response only to the program boundary metadata in the first program (i.e., under calibration thereto), but the program metadata in the second program resets the timer, and the reset timer correctly indicates the position of the program boundary between the terminated first program and the beginning of the second program (as the position corresponding to the "0" count of the reset timer).

[0178] The fourth bitstream (labeled “Scenario 4”) shows a truncated first audio program (P1) containing program boundary metadata (program boundary flag F), and a truncated second audio program (P2) containing program boundary metadata (program boundary flag F), which is spliced ​​with a portion (untruncate) of the first audio program. The program boundary flags at the beginning of the entire second program (before truncation) (some of which are shown in Figure 11) are the same as or similar to those described with reference to Figure 9, and the program boundary flags at the end of the entire first program (some of which are shown in Figure 11) are the same as or similar to those described with reference to Figure 8. The splicing removes the last “N” frames of the first program (and thus some of the program boundary flags that were included before splicing) and the first “M” frames of the second program (and thus some of the program boundary flags that were included before splicing). In a typical embodiment, the encoder or decoder implements a timer (calibrated by a flag in the terminated first program) that counts down towards the end of the first unterminated program, and the same timer (calibrated by a flag in the terminated second program) counts up from the beginning of the unterminated second program. As shown by the boundary timer graph in Scenario 4 of Figure 11, the countdown of such timers (calibrated by program boundary metadata in the first program) is reset (in response to program boundary metadata in the second program) before it reaches 0 (in response to program boundary metadata in the first program). Termination of the first program (by splicing) prevents the timer from identifying the program boundary between the beginning of the terminated first program and the terminated second program in response only to the program boundary metadata in the first program (i.e., under calibration by it).However, a reset timer does not correctly indicate the location of the program boundary between the end of the first program that was terminated and the beginning of the second program that was terminated. Thus, termination of both bitstreams being joined can prevent the precise determination of the boundary between them.

[0179] Embodiments of the present invention may be implemented in hardware, firmware, or software, or a combination thereof (for example, as a programmable logic array). Unless otherwise noted, the algorithms or processes included as part of the present invention are not inherently related to any particular computer or other device. In particular, various general-purpose machines may be used with programs written in accordance with the teachings of this application, or it may be more convenient to construct a more specialized device (for example, an integrated circuit) to perform the required method steps. Thus, the present invention may be implemented in one or more computer programs running on one or more programmable computer systems (for example, an implementation of any of the elements of Figure 1 or the encoder 100 (or its elements) of Figure 2 or the decoder 200 (or its elements) of Figure 3 or the post-processor (or its elements) of Figure 3). Each computer system has at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0180] Each such program may be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, or object-oriented programming languages) for communication with the computer system. In either case, the language may be a compiled language or an interpreted language.

[0181] For example, when implemented by a sequence of computer software instructions, various functions and stages of the embodiments of the present invention may be implemented by a multithreaded software instruction sequence executed in suitable digital signal processing hardware, in which case the various devices, stages and functions of the embodiments may correspond to parts of the software instructions.

[0182] Each such computer program is preferably stored or downloaded to a storage medium or device (e.g., semiconductor memory or media or magnetic or optical media) readable by a general-purpose or dedicated programmable computer, and the computer is configured or operated to perform the procedures described herein when the storage medium or device is read by the computer system. The system of the present invention may be implemented as a computer-readable storage medium configured with (i.e., storing) computer programs, which is configured to operate the computer system in a specific, predefined manner to perform the functions described herein.

[0183] While several embodiments of the present invention have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. In light of the above teachings, numerous modifications and variations of the invention are possible. It will be understood that, within the scope of the appended claims, the invention may be carried out in ways other than those specifically described herein.

[0184] Several aspects are described below. [Aspect 1] It is an audio processing unit: A buffer memory for storing at least one frame of an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and a metadata container, and the metadata container includes a header, one or more metadata payloads and protection data; An audio decoder for decoding the audio data is coupled to the buffer memory; The system has a parser that parses the encoded audio bitstream, which is coupled to or integrated with the audio decoder. The header includes a synchronization word that identifies the beginning of the metadata container, the one or more metadata payloads describe an audio program associated with the audio data, the protection data is located after the one or more metadata payloads, and the protection data can be used to verify the integrity of the metadata container and the one or more payloads within the metadata container. Audio processing unit. [Aspect 2] The audio processing unit according to embodiment 1, wherein the metadata container is stored in a reserved data space of AC-3 or E-AC-3 selected from the group consisting of skip fields, auxiliary data fields, addbsi fields, and combinations thereof. [Aspect 3] The audio processing unit according to embodiment 1 or 2, wherein one or more metadata payloads include metadata indicating at least one boundary between consecutive audio programs. [Aspect 4] The audio processing unit according to embodiment 1 or 2, wherein one or more metadata payloads include a program loudness payload containing data indicating the measured loudness of an audio program. [Aspect 5] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field indicating whether the audio channel contains spoken dialogue. [Aspect 6] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field indicating a loudness measurement method used to generate the loudness data contained in the program loudness payload. [Aspect 7] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using dialogue gating. [Aspect 8] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using an infinite lookahead or file-based loudness correction process. [Aspect 9] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field that indicates the integrated loudness of the audio program without any gain adjustment that can be reduced to dynamic range compression. [Aspect 10] The audio processing unit according to embodiment 4, wherein the program loudness payload includes a field indicating the integrated loudness of the audio program without any gain adjustment that can be reduced to dialogue normalization. [Aspect 11] The audio processing unit according to embodiment 4, configured to perform adaptive loudness processing using the program loudness payload. [Aspect 12] The audio processing unit according to any one of embodiments 1 to 11, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream. [Aspect 13] An audio processing unit according to any one of embodiments 4 to 11, configured to extract the program loudness payload from the encoded audio bitstream and to authenticate or validate the program loudness payload. [Aspect 14] The audio processing unit according to any one of embodiments 1 to 13, wherein each of the one or more metadata payloads includes a unique payload identifier, and the unique payload identifier is located at the beginning of each metadata payload. [Aspect 15] The audio processing unit according to any one of embodiments 1 to 13, wherein the synchronization word is a 16-bit synchronization word having the value 0x5838. [Aspect 16] A method for decoding an encoded audio bitstream: The process involves receiving an encoded audio bitstream that has been segmented into one or more frames; A step of extracting an audio data and metadata container from the encoded audio bitstream, wherein the metadata container includes a header, followed by one or more metadata payloads, and followed by protected data; The steps include verifying the integrity of the container and the one or more metadata payloads through the use of the protected data, The one or more metadata payloads include a program loudness payload that includes data indicating the measured loudness of an audio program associated with the audio data. method. [Aspect 17] The method according to embodiment 16, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream. [Aspect 18] The method according to embodiment 16, further comprising the step of performing adaptive loudness processing on audio data extracted from the encoded audio bitstream using the program loudness payload. [Aspect 19] The method according to embodiment 16, wherein the container is located in and extracted from a reserved data space of AC-3 or E-AC-3 selected from the group consisting of skip fields, auxiliary data fields, addbsi fields and combinations thereof. [Aspect 20] The method according to embodiment 16, wherein the program loudness payload includes a field indicating whether the audio channel contains spoken dialogue. [Aspect 21] The method according to embodiment 16, wherein the program loudness payload includes a field indicating a loudness measurement method used to generate the loudness data contained in the program loudness payload. [Aspect 22] The method according to embodiment 16, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using dialogue gating. [Aspect 23] The method according to embodiment 16, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using an infinite lookahead or file-based loudness correction process. [Aspect 24] The method according to embodiment 16, wherein the program loudness payload includes a field that shows the integrated loudness of the audio program without any gain adjustment resulting from dynamic range compression. [Aspect 25] The method according to embodiment 16, wherein the program loudness payload includes a field indicating the integrated loudness of the audio program without any gain adjustment that can be reduced to dialogue normalization. [Aspect 26] The method according to embodiment 16, wherein the metadata container includes metadata indicating at least one boundary between consecutive audio programs. [Aspect 27] The method according to embodiment 16, wherein the metadata container is stored in one or more skip fields or extra bit segments of a frame.

Claims

1. It is an audio processing unit: A buffer memory configured to store an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata, and the metadata includes a payload of loudness metadata; A parser coupled to the buffer memory and configured to extract the audio data and loudness metadata payloads from the encoded audio bitstream; A decoder coupled to the parser, configured to decode the audio data and generate decoded audio data; The system includes a subsystem coupled to the parser and decoder, configured to receive a target loudness value and to perform processing on the decoded audio data in response to the loudness metadata and the target loudness value. The loudness metadata includes metadata indicating that the payload of the loudness metadata contains short-term loudness of the audio program, and when the payload of the loudness metadata contains short-term loudness of the audio program, the short-term loudness indicates the loudness determined using the measurement method defined in ITU-R BS.1771-1. Audio processing unit.

2. Audio processing method: A step of receiving an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata, and the metadata includes a payload of loudness metadata; The steps include: extracting the audio data and the loudness metadata payload from the encoded audio bitstream; The steps include: decoding the aforementioned audio data and generating the decoded audio data; The stage of receiving the target loudness value; The process includes the step of performing processing on the decoded audio data in response to the loudness metadata and the target loudness value, The loudness metadata includes metadata indicating that the payload of the loudness metadata contains short-term loudness of the audio program, and when the payload of the loudness metadata contains short-term loudness of the audio program, the short-term loudness indicates the loudness determined using the measurement method defined in ITU-R BS.1771-1. Audio processing methods.

3. A non-temporary medium storing software, wherein the software is: A step of receiving an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata, and the metadata includes a payload of loudness metadata; The steps include: extracting the audio data and the loudness metadata payload from the encoded audio bitstream; The steps include: decoding the aforementioned audio data and generating the decoded audio data; The stage of receiving the target loudness value; The instructions include a step of controlling one or more devices to perform a step of processing the decoded audio data in response to the loudness metadata and the target loudness value, The loudness metadata includes metadata indicating that the payload of the loudness metadata contains short-term loudness of the audio program, and when the payload of the loudness metadata contains short-term loudness of the audio program, the short-term loudness indicates the loudness determined using the measurement method defined in ITU-R BS.1771-1. Non-temporary medium.

4. A computer program for causing a computer to perform the method described in claim 1.

Images