Program loudness based on transmission-independent representations

Abstract

Program loudness based on transmission independent representations is disclosed. The present disclosure falls in the field of audio coding, and specifically relates to the field of providing a framework for providing loudness consistency between different audio output signals. In particular, the present disclosure relates to a method, computer program product and apparatus for encoding and decoding an audio data bitstream in order to achieve a desired loudness level of an output audio signal.

Description

[0001] This application is a divisional application filed again for the divisional application with application number 202011037639.9 (the application number of the parent application is 201580054844.7).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 062,479, filed on October 10, 2014, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present invention relates to audio signal processing, and more particularly to encoding and decoding of audio data bit streams in order to achieve a desired loudness level of an output audio signal. Background Art

[0005] Dolby AC-4 is an audio format for efficiently distributing rich media content. AC-4 provides broadcasters and content producers with a flexible framework for distributing and encoding content in an efficient manner. Content can be distributed over several substreams, such as M&E (music and effects) in one substream and dialogue in a second substream. For some audio content, it may be advantageous, for example, to switch the language of the dialogue from one language to another, or to be able to add, for example, a commentary substream to the content or an additional substream including descriptions for the visually impaired.

[0006] In order to ensure proper leveling of the content presented to the consumer, the loudness of the content needs to be known with some accuracy. Current loudness requirements have a tolerance of 2dB (ATSC A / 85), 0.5dB (EBU R128), while some specifications have a tolerance as low as 0.1dB. This means that the loudness of an output audio signal with a commentary track and with dialogue in a first language should be substantially the same as the loudness of an output audio signal without a commentary track, but with dialogue in a second language. Summary of the invention

[0007] The present invention provides a method of processing a bitstream comprising a plurality of content substreams, each content substream representing an audio signal, the method comprising: extracting one or more presentation data structures from the bitstream, each presentation data structure comprising a reference to at least one of the content substreams, each presentation data structure further comprising a reference to a metadata substream, the metadata substream representing loudness data describing a combination of the referenced one or more content substreams; receiving data indicating a selected one of the one or more presentation data structures and a desired loudness level; decoding the one or more content substreams referenced by the selected presentation data structure; and forming an output audio signal based on the decoded content substreams, the method further comprising processing the decoded one or more content substreams or the output audio signal to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure.

[0008] The present invention also provides a decoder for processing a bitstream comprising a plurality of content substreams, each content substream representing an audio signal, the decoder comprising: a receiving component configured to receive the bitstream; a demultiplexer configured to extract one or more presentation data structures from the bitstream, each presentation data structure comprising a reference to at least one of the content substreams and further comprising a reference to a metadata substream representing loudness data describing a combination of the referenced one or more content substreams; a playback state component configured to receive data indicating a selected presentation data structure among the one or more presentation data structures and a desired loudness level; and a mixing component configured to decode the one or more content substreams referenced by the selected presentation data structure and form an output audio signal based on the decoded content substreams, wherein the mixing component is further configured to process the decoded one or more content substreams or the output audio signal to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure.

[0009] The present invention also provides an audio encoding method, comprising: receiving a plurality of content substreams representing respective audio signals; defining one or more presentation data structures, each presentation data structure referencing at least one of the plurality of content substreams; for each of the one or more presentation data structures, applying a predefined loudness function to obtain loudness data describing a combination of the referenced one or more content substreams and including a reference to the loudness data from the presentation data structure; and forming a bitstream, the bitstream comprising the plurality of content substreams, the one or more presentation data structures, and the loudness data referenced by the presentation data structures.

[0010] The present invention also provides an audio encoder, comprising: a loudness component configured to apply a predefined loudness function to obtain loudness data, the loudness data describing a combination of one or more content substreams representing a corresponding audio signal; a presentation data component configured to define one or more presentation data structures, each presentation data structure comprising a reference to one or more content substreams among a plurality of content substreams and a reference to loudness data describing the combination of the referenced content substreams; and a multiplexing component configured to form a bitstream, the bitstream comprising the plurality of content substreams, the one or more presentation data structures and the loudness data referenced by the presentation data structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Example embodiments will now be described with reference to the accompanying drawings, in which:

[0012] Figure 1 is a generalized block diagram illustrating a decoder for processing a bitstream and achieving a desired loudness level of an output audio signal;

[0013] Figure 2 yes Figure 1 A generalized block diagram of a first embodiment of a mixing component of a decoder;

[0014] Figure 3 yes Figure 1 A generalized block diagram of a second embodiment of a mixing component of a decoder;

[0015] Figure 4 describing a presentation data structure according to an embodiment;

[0016] Figure 5 shows a generalized block diagram of an audio encoder according to an embodiment; and

[0017] Figure 6 Description by Figure 5 The bit stream formed by the audio encoder.

[0018] All the figures are schematic and generally only show parts which are necessary for elucidating the disclosure, while other parts may be omitted or merely suggested. Unless otherwise indicated, the same reference numerals refer to the same parts in the different figures. DETAILED DESCRIPTION

[0019] In view of the above, it is an object to provide an encoder and decoder and associated methods which aim to provide a desired loudness level for an output audio signal independently of what content substreams are mixed into the output audio signal.

[0020] I. Overview - Decoder

[0021] According to a first aspect, example embodiments propose a decoding method, a decoder and a computer program product for decoding. The proposed method, decoder and computer program product may generally have the same features and advantages.

[0022] According to an example embodiment, there is provided a method of processing a bitstream comprising a plurality of content substreams, each content substream representing an audio signal, the method comprising: extracting one or more presentation data structures from the bitstream, each presentation data structure comprising a reference to at least one of the content substreams, each presentation data structure further comprising a reference to a metadata substream representing loudness data describing a combination of the referenced one or more content substreams; receiving data indicative of a selected one of the one or more presentation data structures and a desired loudness level; decoding the one or more content substreams referenced by the selected presentation data structure; and forming an output audio signal based on the decoded content substreams, the method further comprising processing the decoded one or more content substreams or the output audio signal to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure.

[0023] The data indicating the selected presentation data structure and the desired loudness level are typically user settings available at the decoder. The user can, for example, use a remote control to select a presentation data structure, where the dialogue is in French, and / or to increase or decrease the desired output loudness level. In many embodiments, the output loudness level is related to the capabilities of the playback device. According to some embodiments, the output loudness level is controlled by a volume. Therefore, the data indicating the selected presentation data structure and the desired loudness value are typically not included in the bitstream received by the decoder.

[0024] As used herein, "loudness" refers to a modeled psychoacoustic measure of sound intensity; in other words, loudness represents an approximation of the volume of a sound or sounds as perceived by an average user.

[0025] As used herein, "loudness data" refers to data resulting from a measurement of the loudness level of a particular presentation data structure using a function that models psychoacoustic loudness perception. In other words, it is a set of values ​​that indicate a loudness property of a combination of one or more content substreams referenced by the particular presentation data structure. According to an embodiment, the average loudness level of the combination of one or more content substreams referenced by the particular presentation data structure may be measured. For example, the loudness data may refer to a dialnorm value (according to ITU-R BS.1770 recommendation) for one or more content substreams referenced by the particular presentation data structure. Other suitable loudness measurement standards may be used, such as the Glasberg and Moore loudness model that provides a modification and extension of the Zwicker loudness model.

[0026] As used herein, a "presentation data structure" refers to metadata related to the content of an output audio signal. The output audio signal will also be referred to as a "program." A presentation data structure will also be referred to as a "presentation."

[0027] The audio content may be distributed over several substreams. As used herein, a "content substream" refers to such a substream. For example, a content substream may include music of the audio content, dialogue of the audio content, or a commentary track to be included in the output audio signal. The content substream may be either channel-based or object-based. In the latter case, time-dependent spatial position data is included in the content substream. The content substream may be included in the bitstream or may be part of the audio signal (i.e., as a channel group or an object group).

[0028] As used herein, an "output audio signal" refers to an actual output audio signal that will be rendered and provided to a user.

[0029] The inventors have realised that by providing loudness data, eg a dialog norm value, for each presentation, specific loudness data indicating exactly what loudness to use for the referenced at least one content substream when decoding that particular presentation may be made available to a decoder.

[0030] In the prior art, loudness data may be provided for each content substream. A problem with providing loudness data for each content substream is that in this case it is up to the decoder to combine the various loudness data into the loudness of the presentation. Adding the individual loudness data values ​​of the substreams (which represent the average loudness of the substreams) to arrive at a loudness value for a certain presentation may be inaccurate and in many cases will not result in the actual average loudness value of the combined substreams. Due to the nature of the signals, loudness algorithms and the nature of loudness perception (which is generally non-additive), adding the loudness data for each referenced content substream may be mathematically impossible and may result in potential inaccuracies that are larger than the tolerances indicated above.

[0031] Using the present embodiment, the difference between the average loudness level of the selected presentation and the desired loudness level provided by the loudness data for the selected presentation may therefore be used to control the playback gain of the output audio signal.

[0032] By providing and using loudness data as described above, consistent loudness, i.e., loudness close to the desired loudness level, can be achieved between different presentations. In addition, consistent loudness can be achieved between different programs on a television channel (e.g., between a television program and its commercials), and across television channels.

[0033] According to an example embodiment, wherein the selected presentation data structure references two or more content substreams and also references at least two mixing coefficients to be applied to the content substreams, said forming the output audio signal further comprises additively mixing the decoded one or more content substreams by applying the mixing coefficient(s).

[0034] By providing at least two mixing coefficients, an increased flexibility of the content of the output audio signal is achieved.

[0035] For example, for each of the two or more content substreams, the selected presentation data structure may reference one mixing coefficient to be applied to the respective substream. According to this embodiment, the relative loudness levels between the content substreams may be varied. For example, cultural preferences may require a different balance between different content substreams. Consider the case where the Spanish region wants less attention to music. Therefore, the music substream is attenuated by 3 dB. According to other embodiments, a single mixing coefficient may be applied to a subset of two or more content substreams.

[0036] According to an example embodiment, the bitstream comprises a plurality of time frames, and wherein the mixing coefficients referenced by the selected presentation data structure are independently assignable to each time frame. An effect of providing time-varying mixing coefficients is that ducking may be achieved. For example, the loudness level of a time segment for one content substream may be reduced by an increased loudness in the same time segment of another content substream.

[0037] According to an example embodiment, the loudness data represents a value of a loudness function associated with applying a gating to its audio input signal.

[0038] The audio input signal is the signal at the encoder end to which a loudness function (i.e., a dialog normalization function) is applied. The resulting loudness data is then sent to the decoder in a bitstream. A noise gate (also known as a mute gate) is an electronic device or software used to control the volume of an audio signal. Gating is the use of such a gate. A noise gate attenuates a registered signal below a threshold. A noise gate can attenuate a signal by a fixed amount, which is called a range. In its simplest form, a noise gate allows a signal to pass only if it is above a set threshold.

[0039] Gating may also be based on the presence of dialogue in the audio input signal. Thus, according to example embodiments, the loudness data represents the value of a loudness function associated with such time periods of its audio input signal that represent dialogue. According to other embodiments, gating is based on a minimum loudness level. Such a minimum loudness level may be an absolute threshold or a relative threshold. A relative threshold may be based on a loudness level measured with an absolute threshold.

[0040] According to an example embodiment, the presentation data structure further comprises a reference to dynamic range compression (DRC) data for the referenced one or more content substreams, the method further comprising processing the decoded one or more content substreams or the output audio signal based on the DRC data, wherein the processing comprises applying one or more DRC gains to the decoded one or more content substreams or the output audio signal.

[0041] Dynamic range compression reduces the volume of loud sounds or amplifies quiet sounds, thereby narrowing or "compressing" the dynamic range of the audio signal. By providing DRC data uniquely for each presentation, an improved user experience of the output audio signal can be achieved regardless of what presentation is selected. Moreover, by providing DRC data for each presentation, a consistent user experience of the audio output signal can be achieved across multiple presentations, as described above, and also between programs, and across television channels.

[0042] DRC gains are always time-varying. In each time period, the DRC gain can be a single gain for the audio output signal, or a different DRC gain for each substream. The DRC gain can be applied to multiple groups of channels and / or be frequency-dependent. In addition, the DRC gain included in the DRC data can represent the DRC gain for two or more DRC time periods (e.g., subframes of a time frame defined by an encoder).

[0043] According to an example embodiment, the DRC data includes at least one set of one or more DRC gains. The DRC data may therefore include a plurality of DRC profiles corresponding to the DRC modes, each DRC profile providing a different user experience of the audio output signal. By directly including the DRC gains in the DRC data, a reduced computational complexity of the decoder may be achieved.

[0044] According to an example embodiment, the DRC data comprises at least one compression curve, and wherein the one or more DRC gains are obtained by calculating one or more loudness values ​​of the one or more content substreams or the audio output signal using a predefined loudness function, and mapping the one or more loudness values ​​to the DRC gains using the compression curve. By providing the compression curves in the DRC data and calculating the DRC gains based on these curves, the bit rate required for sending the DRC data to the encoder may be reduced. The predefined loudness function may be taken, for example, from the ITU-R BS.1770 recommendation document, but any suitable loudness function may be used.

[0045] According to an example embodiment, the mapping of loudness values ​​includes a smoothing operation of the DRC gains. The effect of this may be a better perceived output audio signal. A time constant for smoothing the DRC gains may be sent as part of the DRC data. Such time constants may be different depending on the signal properties. For example, in some embodiments, when the loudness value is greater than a previous corresponding loudness value, the time constant may be smaller than when the loudness value is less than a previous corresponding loudness value.

[0046] According to an example embodiment, the referenced DRC data is included in the metadata substream. This may reduce the decoding complexity of the bitstream.

[0047] According to an example embodiment, each of the decoded one or more content substreams comprises substream-level loudness data describing a loudness level of the content substream, and wherein said processing of the decoded one or more content substreams or the output audio signal further comprises ensuring that loudness consistency is provided based on the loudness levels of the content substreams.

[0048] As used herein, "loudness consistency" means that the loudness is consistent between different presentations, i.e., consistent on output audio signals formed based on different content substreams. Moreover, the term means that the loudness is consistent between different programs, i.e., between completely different output audio signals (such as the audio signal of a television program and the audio signal of a commercial advertisement). Furthermore, the term means that the loudness is consistent across different television channels.

[0049] Providing loudness data describing the loudness levels of content substreams may help a decoder provide loudness consistency in some cases. For example, in cases where the forming of the output audio signal comprises combining two or more decoded content substreams using alternative mixing coefficients, and where the loudness data is compensated using substream-level loudness data for providing loudness consistency. These alternative mixing coefficients may be derived from user input, for example, in cases where a user decides to deviate from the default presentation (e.g., through dialogue enhancement, dialogue attenuation, scene personalization, etc.). This may compromise loudness compliance, as user influence may cause the loudness of the audio output signal to fall outside of compliance rules. To assist with loudness consistency in these cases, the present embodiment provides the option of sending substream-level loudness data.

[0050] According to some embodiments, the reference to at least one of the content substreams is a reference to at least one content substream group consisting of one or more of the content substreams. This can reduce the complexity of the decoder, because multiple presentations can share a content substream group (e.g., a substream group consists of a content substream related to music and a content substream related to effects). This can also reduce the bit rate required to send the bitstream.

[0051] According to some embodiments, for a content substream group, the selected presentation data structure references a single mixing coefficient to be applied to each of the one or more of the content substreams constituting the content substream group.

[0052] This may be advantageous in the case where the loudness levels of the content substreams in a content substream group are good relative to each other, but the overall loudness level of the content substreams in the content substream group should be increased or decreased compared to the other content substream(s) or content substream group(s) referenced by the selected presentation data structure.

[0053] According to some embodiments, the bitstream comprises a plurality of time frames, and wherein data indicating a selected presentation data structure among one or more presentation data structures is independently assignable to each time frame. Thus, in the event that a plurality of presentation data structures are received for a program, the selected presentation data structure may be changed, for example, by a user, while the program is ongoing. Thus, the present embodiment provides a more flexible way of selecting the content of the output audio while providing loudness consistency of the output audio signal.

[0054] According to some embodiments, the method further comprises: extracting one or more presentation data structures from the bitstream for the first frame of the plurality of time frames, and extracting one or more presentation data structures different from the one or more presentation data structures extracted from the first frame of the plurality of time frames from the bitstream for the second frame of the plurality of time frames, and wherein the data indicating the selected presentation data structure indicates the selected presentation data structure for the time frame to which it is assigned. Thus, a plurality of presentation data structures may be received in the bitstream, wherein, wherein, some of the presentation data structures are associated with the first group of time frames, and some of the presentation data structures are associated with the second group of time frames. For example, the commentary track may be available only for a certain time period of the program. Moreover, the presentation data structure currently applicable at a particular point in time may be used to select the selected presentation data structure while the program is in progress. Therefore, the present embodiment provides a more flexible way of selecting the content of the output audio while providing loudness consistency of the output audio signal.

[0055] According to some embodiments, among a plurality of content substreams included in a bitstream, only one or more content substreams referenced by a selected presentation data structure are decoded. This embodiment may provide a highly efficient decoder with reduced computational complexity.

[0056] According to some embodiments, the bitstream includes two or more separate bitstreams, each separate bitstream includes at least one of the multiple content substreams, wherein the step of decoding one or more content substreams referenced by the selected presentation data structure includes: for each specific bitstream in the two or more separate bitstreams, (one or more) content substreams among the referenced content substreams included in the specific bitstream are separately decoded. According to this embodiment, each separate bitstream can be received by a separate decoder, which decodes the (one or more) content substreams required according to the selected presentation structure provided in the separate bitstream. This can improve the decoding speed because the separate decoders can work in parallel. Therefore, the decoding performed by the separate decoders can at least partially overlap. However, it should be noted that the decoding performed by the separate decoders does not need to overlap.

[0057] Furthermore, by splitting the content substream into several bitstreams, the present embodiment enables at least two separate bitstreams to be received over different infrastructures as described below.Thus, the present embodiment provides a more flexible way of receiving multiple content substreams at a decoder.

[0058] Each decoder may process (one or more) decoded substreams based on the loudness data referenced by the selected presentation data structure, and / or apply DRC gains, and / or apply mixing coefficients to (one or more) decoded substreams. The processed or unprocessed content substreams may then be provided from all decoders of at least two decoders to a mixing component for forming an output audio signal. Alternatively, the mixing component performs loudness processing and / or applies DRC gains and / or applies mixing coefficients. In some embodiments, a first decoder may receive a first bitstream of two or more separate bitstreams via a first infrastructure (e.g., cable television broadcast), while a second decoder receives a second bitstream of two or more separate bitstreams via a second infrastructure (e.g., via the Internet). According to some embodiments, the one or more presentation data structures are present in all bitstreams of the two or more separate bitstreams. In this case, the presentation definition and loudness data are present in all separate decoders. This makes it possible to operate the decoders independently up to the mixing component. References to substreams that are not present in the corresponding bitstream may be indicated as being provided externally.

[0059] According to an example embodiment, there is provided a decoder for processing a bitstream comprising a plurality of content substreams, each content substream representing an audio signal, the decoder comprising: a receiving component configured to receive the bitstream; a demultiplexer configured to extract one or more presentation data structures from the bitstream, each presentation data structure comprising a reference to at least one of the content substreams and further comprising a reference to a metadata substream representing loudness data describing a combination of the referenced one or more content substreams; a playback state component configured to receive data indicating a selected presentation data structure among the one or more presentation data structures and a desired loudness level; and a mixing component configured to decode the one or more content substreams referenced by the selected presentation data structure and to form an output audio signal based on the decoded content substreams, wherein the mixing component is further configured to process the decoded one or more content substreams or the output audio signal to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure.

[0060] II. Overview - Encoder

[0061] According to a second aspect, example embodiments propose a coding method, an encoder and a computer program product for coding. The proposed method, encoder and computer program product may generally have the same features and advantages. Generally, the features of the second aspect may have the same advantages as the corresponding features of the first aspect.

[0062] According to an example embodiment, there is provided an audio encoding method comprising: receiving a plurality of content substreams representing respective audio signals; defining one or more presentation data structures, each presentation data structure referencing at least one of the plurality of content substreams; for each of the one or more presentation data structures, applying a predefined loudness function to obtain loudness data describing a combination of the referenced one or more content substreams and including a reference to the loudness data from the presentation data structure; and forming a bitstream comprising the plurality of content substreams, the one or more presentation data structures, and the loudness data referenced by the presentation data structures.

[0063] As mentioned above, the term "content substream" includes substreams both within the bitstream and within the audio signal. An audio encoder typically receives audio signals, which are then encoded into a bitstream. The audio signal can be grouped, wherein each group can be characterized as a separate encoder input audio signal. Each group can then be encoded as a substream.

[0064] According to some embodiments, the method further comprises the steps of: for each of the one or more presentation data structures, determining dynamic range compression DRC data for the referenced one or more content substreams, wherein the DRC data quantizes at least one desired compression curve or at least one set of DRC gains; and including the DRC data in the bitstream.

[0065] According to some embodiments, the method further comprises the steps of: for each of the plurality of content substreams, applying a predefined loudness function to obtain substream-horizontal loudness data for the content substream; and including the substream-horizontal loudness data in the bitstream.

[0066] According to some embodiments, the predefined loudness function is related to applying gating to the audio signal.

[0067] According to some embodiments, the predefined loudness function is only relevant to such time periods of the audio signal representing a conversation.

[0068] According to some embodiments, the predefined loudness function comprises at least one of: frequency-dependent weighting of the audio signal, channel-dependent weighting of the audio signal, disregarding segments of the audio signal having a signal power below a threshold; calculating an energy measure of the audio signal.

[0069] According to an example embodiment, there is provided an audio encoder comprising: a loudness component configured to apply a predefined loudness function to obtain loudness data describing a combination of one or more content substreams representing a corresponding audio signal; a presentation data component configured to define one or more presentation data structures, each presentation data structure comprising a reference to one or more content substreams among a plurality of content substreams and a reference to loudness data describing the combination of the referenced content substreams; and a multiplexing component configured to form a bitstream comprising the plurality of content substreams, the one or more presentation data structures and the loudness data referenced by the presentation data structures.

[0070] III. Example Embodiments

[0071] Figure 1 A generalized block diagram of a decoder 100 for processing a bitstream P and achieving a desired loudness level of an output audio signal 114 is illustrated.

[0072] The decoder 100 comprises receiving means (not shown) configured to receive a bitstream P comprising a plurality of content substreams, each content substream representing an audio signal.

[0073] The decoder 100 also includes a demultiplexer 102 configured to extract one or more presentation data structures 104 from the bitstream P. Each presentation data structure includes a reference to at least one of the content substreams. In other words, a presentation data structure or presentation is a description of which content substreams are to be combined. As noted above, content substreams encoded in two or more separate substreams can be combined into one presentation.

[0074] Each presentation data structure also includes a reference to a metadata substream representing loudness data describing a combination of the referenced one or more content substreams.

[0075] Now combine Figure 4 To describe the contents of the presentation data structure and its different references.

[0076] exist Figure 4 , different sub-streams 412, 205 that may be referenced by the extracted one or more presentation data structures 104 are shown. Among the three presentation data structures 104, the selected presentation data structure 110 is selected. Figure 4 Clearly, the bitstream P includes content substreams 412, metadata substreams 205, and the one or more presentation data structures 104. The content substreams 412 may, for example, include a substream for music, a substream for effects, a substream for ambience, a substream for English dialogue, a substream for Spanish dialogue, a substream for associated audio (AA) in English (e.g., an English commentary track), and a substream for AA in Spanish (e.g., a Spanish commentary track).

[0077] exist Figure 4 In the embodiment of the present invention, all content substreams 412 are encoded in the same bitstream P, but as noted above, this is not always the case. Broadcasters of audio content can use a single bitstream configuration (e.g., a single packet identifier (PID) configuration in the MPEG standard or a multiple bitstream configuration (e.g., a dual PID configuration)) to send the audio content to their clients, i.e., decoders.

[0078] The present disclosure introduces an intermediate level in the form of substream groups that reside between the presentation layer and the substream layer. A content substream group can group or reference one or more content substreams. A presentation can then reference a content substream group. Figure 4 In , the content substreams music, effects and ambience are grouped to form a content substream group 410 referenced 404 by the selected presentation data structure 110 .

[0079] Content substream groups provide greater flexibility in combining content substreams. Specifically, the substream group level provides a means to collect or group several content substreams into a unique group (eg, content substream group 410 including music, effects, and ambience).

[0080] This may be advantageous because a content substream group (e.g., for music and effects or for music, effects and ambience) may be used for more than one presentation, such as a presentation combined with English or Spanish dialogue. Similarly, a content substream may also be used in more than one content substream group.

[0081] Furthermore, depending on the syntax of the presentation data structure, using content substream groups may provide the possibility to mix a large number of content substreams for presentation.

[0082] According to some embodiments, a presentation 104, 110 will always consist of one or more sub-stream groups.

[0083] Figure 4 The selected presentation data structure 110 in the plurality of presentation data structures 104 includes a reference 404 to a content substream group 410, which consists of one or more of the content substreams. The selected presentation data structure 110 also includes a reference to a content substream for Spanish dialogue and a reference to a content substream for AA in Spanish. Moreover, the selected presentation data structure 110 includes a reference 406 to a metadata substream 205, which represents loudness data 408 describing a combination of the referenced one or more content substreams. It is clear that the other two presentation data structures of the plurality of presentation data structures 104 may include similar data as the selected presentation data structure 110. According to other embodiments, the bitstream P may include additional metadata substreams similar to the metadata substream 205, wherein these additional metadata substreams are referenced from other presentation data structures. In other words, each presentation data structure in the plurality of presentation data structures 104 may reference dedicated loudness data.

[0084] The selected presentation data structure may change over time, i.e., if the user decides to turn off the Spanish commentary track AA(ES). In other words, the bitstream P comprises a plurality of time frames, and wherein data ( Figure 1 Reference 108) in can be assigned to each time frame independently.

[0085] As described above, the bitstream P includes a plurality of time frames. According to some embodiments, one or more presentation data structures 104 may be associated with different time periods of the bitstream P. In other words, the demultiplexer ( Figure 1102 in the figure) may be configured to extract one or more presentation data structures for a first frame of the plurality of time frames from the bitstream P, and further configured to extract one or more presentation data structures for a second frame of the plurality of time frames from the bitstream P that are different from the one or more presentation data structures extracted from the first frame of the plurality of time frames. In this case, the data indicating the selected presentation data structure ( Figure 1 Reference numeral 108 in ) indicates a presentation data structure for selection of the time frame to which it is assigned.

[0086] Now return to Figure 1 , the decoder 100 also includes a playback state component 106. The playback state component 106 is configured to receive data 108, which indicates a selected presentation data structure 110 among the one or more presentation data structures 104. The data 108 also includes an expected loudness level. As mentioned above, the data 108 can be provided by a consumer of the audio content to be decoded by the decoder 100. The expected loudness value can also be a decoder-specific setting, which depends on the playback device that will be used to playback the output audio signal. The consumer can, for example, select that the audio content should include Spanish dialogue as understood from above.

[0087] The decoder 100 also includes a mixing component that receives the selected presentation data structure 110 from the playback state component 106 and decodes one or more content substreams referenced by the selected presentation data structure 110 from the bitstream P. According to some embodiments, only the one or more content substreams referenced by the selected presentation data structure 110 are decoded by the mixing component. Thus, in the event that the consumer has selected a presentation using, for example, Spanish dialogue, any content substream representing English dialogue will not be decoded, which reduces the computational complexity of the decoder 100.

[0088] The mixing component 112 is configured to form an output audio signal 114 based on the decoded content substreams.

[0089] Furthermore, the mixing component 112 is configured to process the decoded one or more content substreams or output audio signals to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure 110 .

[0090] Figure 2 and Figure 3 Various embodiments of the mixing element 112 are described.

[0091] exist Figure 2, a bitstream P is received by a substream decoding component 202, which decodes one or more content substreams 204 referenced by a selected presentation data structure 110 from the bitstream P based on the selected presentation data structure 110. The one or more decoded content substreams 204 are then sent to a component 206, which is used to form an output audio signal 114 based on the decoded content substreams 204 and the metadata substream 205. When forming the audio output signal, the component 206 may, for example, take into account any time-related spatial position data included in the (one or more) content substreams 204. The component 206 may also take into account DRC data included in the metadata substream 205. Alternatively, a loudness component 210 (described below) processes the output audio signal 114 based on the DRC data. In some embodiments, the component 206 receives the content substream 110 from the presentation data structure 110 ( Figure 2 1 (not shown) and the mixing coefficients (described below) are calculated and applied to the corresponding content substream 204. The output audio signal 114* is then sent to the loudness component 210, which processes the output audio signal 114* to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure 110 (which is included in the metadata substream 205) and the desired loudness level included in the data 108, thereby outputting the loudness-processed output audio signal 114.

[0092] exist Figure 3 , a similar mixing element 112 is shown, with Figure 2 The difference between the mixing component 112 described in FIG. 1 and FIG. 2 is that the component 206 for forming the output audio signal and the loudness component 210 have positions that are changed relative to each other. Thus, the loudness component 210 processes the decoded one or more content substreams 204 to reach the desired loudness level (based on the loudness data included in the metadata substream 205) and outputs one or more loudness processed content substreams 204*. These content substreams 204* are then sent to the component 206, which is used to form the output audio signal of the output loudness processed output audio signal 114. As shown in conjunction with Figure 2 As described above, DRC data (which is included in metadata substream 205) can be applied either in component 206 or in loudness component 210. Also, in some embodiments, component 206 receives DRC data from presentation data structure 110 ( Figure 3 204*) and applies these mixing coefficients to the corresponding content substream 204*.

[0093] Each of the one or more presentation data structures 104 includes dedicated loudness data that indicates exactly what the loudness of the content substream referenced by the presentation data structure will be when decoded. The loudness data may, for example, represent a dialog normalization value. According to some embodiments, the loudness data represents the value of a loudness function to which a gating is applied to an audio input signal. This may improve the accuracy of the loudness data. For example, if the loudness data is based on a band-limiting loudness function, the background noise of the audio input signal will not be taken into account when calculating the loudness data, as frequency bands containing only static may be disregarded.

[0094] Furthermore, the loudness data may represent the value of a loudness function associated with such time periods of the audio input signal representing dialogue. This is in accordance with the ARSC A / 85 standard, in which the dialog norm is explicitly defined with respect to the loudness of the dialogue (Anchor element): "The value of the dialog norm parameter indicates the loudness of the Anchor element of the content".

[0095] Processing of one or more decoded content substreams or output audio signals or leveling of output audio signals based on loudness data referenced by the selected presentation data structure to achieve the desired loudness level, i.e., ORL L It can thus be performed using the dialog specification DN(pres) calculated from the presentation above:

[0096] g L =ORL-DN(pres)

[0097] Among them, DN(pres) and ORL are usually expressed in dB. FS Values ​​expressed in dB (referenced to a full-scale 1kHz sine (or square) wave).

[0098] According to some embodiments, where the selected presentation data structure references two or more content substreams, the selected presentation data structure further references at least one mixing coefficient to be applied to the two or more content substreams. The mixing coefficient(s) may be used to provide a modified relative loudness level between the content substreams referenced by the selected presentation. The mixing coefficients may be applied as a wideband gain to a channel / object in a content substream before mixing it with channels / objects in (one or more) other content substreams.

[0099] The at least one mixing coefficient is typically static, but may be independently assignable to each time frame of the bitstream, for example, to implement ducking.

[0100] The mixing coefficients therefore do not need to be sent in the bitstream for every time frame; they can remain valid until overwritten.

[0101] The mixing coefficients may be defined for each content substream. In other words, for each of the two or more substreams, the selected presentation data structure may reference one mixing coefficient to be applied to the corresponding substream.

[0102] According to some embodiments, a mixing coefficient may be defined for each content substream group and applied to all content substreams in the content substream group. In other words, for a content substream group, a selected presentation data structure may reference a single mixing coefficient to be applied to each of the one or more of the content substreams that make up the substream group.

[0103] According to yet another embodiment, the selected presentation data structure may reference a single mixing coefficient to be applied to each of the two or more content substreams.

[0104] Table 1 below indicates an example of object transmission. Objects are clustered in categories distributed over several substreams. All presentation data structures combine music and effects containing the main part of the audio content without dialogue. The combination is therefore a content substream group. Depending on the selected presentation data structure, a certain language is selected, for example, English (D#1) or Spanish D#2. Moreover, the content substreams include an associated audio substream in English (Desc#1) and an associated audio substream in Spanish (Desc#2). The associated audio can include enhanced audio, such as audio description, narrators for the hard of hearing, narrators for the visually impaired, commentary tracks, etc.

[0105]

[0106] In rendering 1, no mixing gain should be applied via the mixing coefficients; rendering 1 therefore does not reference mixing coefficients at all.

[0107] Cultural preferences may require a different balance between categories. This is illustrated in presentation 2. Consider the case where the Spanish region wants to focus less on music. Therefore, the music substream is attenuated by 3dB. In this example, for each of the two or more substreams, presentation 2 references one mixing coefficient to be applied to the corresponding substream.

[0108] Presentation 3 includes a Spanish description stream for the visually impaired. This stream was recorded in a booth and is too loud to be mixed directly into the presentation, so it is attenuated by 6dB. In this example, for each of the two or more substreams, Presentation 3 references one mixing coefficient to be applied to the corresponding substream.

[0109] In presentation 4, both the music substream and the effects substream are attenuated by 3 dB. In this case, for an M&E substream group, presentation 4 references a single mixing coefficient to be applied to each of the one or more of the content substreams making up that M&E substream group.

[0110] According to some embodiments, a user or consumer of audio content may provide user input that causes the output audio signal to deviate from the selected presentation data structure. For example, dialogue enhancement or dialogue attenuation may be requested by the user, or the user may want to perform some scene personalization, such as increasing the volume of an effect. In other words, alternative mixing coefficients may be provided for use when combining two or more decoded content substreams for forming an output audio signal. This may affect the loudness level of the audio output signal. In order to provide loudness consistency in this case, each of the one or more decoded content substreams may include substream-level loudness data describing the loudness level of the content substream. The substream-level loudness data may then be used to compensate the loudness data for providing loudness consistency.

[0111] The substream-level loudness data may be similar to the loudness data referenced by the presentation data structure, and may advantageously represent values ​​of a loudness function, optionally with a larger range to encompass the typically quieter signals in the content substream.

[0112] There are many ways of using this data to achieve loudness consistency. The following algorithm is shown by way of example.

[0113] Let DN(P) be the presentation dialogue specification, DN(S i ) is the substream loudness of substream i.

[0114] If the decoder is based on the reference as a content substream group S M&E Music content substream S M and effect content substream S E , plus the reference conversation content sub-stream S D To form the audio output signal based on the presentation of the content substream, if you want to maintain consistent loudness while applying 9dB of dialogue enhancement DE, the decoder can use DE to predict the new presentation loudness DN (P DE ):

[0115] DN(P DE )=log 10 (10 DN(S M&E ) )+10 DN(S D )+9 )

[0116] As mentioned above, performing such addition of substream loudness when approximating presentation loudness can result in loudness that is very different from the actual loudness. Therefore, an alternative is to calculate the approximation without DE to find the offset from the actual loudness:

[0117] offset=DN(P)-log 10 (10 DN(S M&E ) )+10 DN(S D ))

[0118] Since the gain on DE does not modify the program in a way that the different substream signals interact with each other, it is possible that when this offset is used to adjust DN (P DE ) is corrected, the approximation becomes more accurate:

[0119] DN(P DE )=log 10 (10 DN(S M&E ) ))+10 DN(S D )+9 ))+offset

[0120] According to some embodiments, the presentation data structure also includes a reference to dynamic range compression DRC data for the referenced one or more content substreams 204. The DRC data may be used to process the decoded one or more content substreams 204 by applying one or more DRC gains to the decoded one or more content substreams 204 or the output audio signal 114. The one or more DRC gains may be included in the DRC data, or they may be calculated based on one or more compression curves included in the DRC data. In this case, the decoder 100 uses a predefined loudness function to calculate a loudness value for each of the referenced one or more content substreams 204 or for the output audio signal 114, and then uses the (one or more) loudness values ​​to map to the DRC gains using the (one or more) compression curves. The mapping of the loudness values ​​may include a smoothing operation of the DRC gains.

[0121] According to some embodiments, the DRC data referenced by the presentation data structure corresponds to a plurality of DRC profiles. These DRC profiles are tailored for the particular audio signals to which they can be applied. Profiles can range from no compression ("None"), to fairly mild compression (e.g., "Music Mild"), all the way to extremely aggressive compression (e.g., "Speech"). Thus, the DRC data can include multiple sets of DRC gains or multiple compression curves from which multiple sets of DRC gains can be obtained.

[0122] The referenced DRC data may be included in the Figure 4 In the metadata substream 205.

[0123] It should be noted that the bitstream P may include two or more individual bitstreams according to some embodiments, and the content substreams may be encoded as different bitstreams in this case. One or more presentation data structures are advantageously included in all individual bitstreams in this case, which means that several decoders (one decoder for each individual bitstream) can work individually and completely independently to decode the content substreams referenced by the selected presentation data structure (which is also provided to each individual decoder). According to some embodiments, the decoders may work in parallel. Each individual decoder decodes the substream present in the individual bitstream it receives. According to an embodiment, each individual decoder performs processing of the content substream decoded by it to achieve the desired loudness level. The processed content substream is then provided to another mixing component, which forms an output audio signal with the desired loudness level.

[0124] According to other embodiments, each individual decoder provides its decoded and unprocessed substream to a further mixing component which performs loudness processing and then forms an output audio signal from all of the one or more content substreams referenced by the selected presentation data structure, or first mixes the one or more content substreams and performs loudness processing on the mixed signal. According to other embodiments, each individual decoder performs a mixing operation on two or more of its decoded substreams. The further mixing component then mixes the pre-mixed contributions of the individual decoders.

[0125] Figure 5 Combination Figure 6An audio encoder 500 is shown by way of example. The encoder 500 comprises a presentation data component 504 configured to define one or more presentation data structures 506, each presentation data structure 506 comprising references 604, 605 to one or more content substreams 612 of a plurality of content substreams 502 and a reference 608 to loudness data 510, the loudness data 510 describing a combination of the referenced content substreams 612. The encoder 500 further comprises a loudness component 508 configured to apply a predefined loudness function 514 to obtain loudness data 510, the loudness data 510 describing a combination of the one or more content substreams representing a corresponding audio signal. The encoder further comprises a multiplexing component 512 configured to form a bitstream P comprising the plurality of content substreams, the one or more presentation data structures 506 and the loudness data 510 referenced by the one or more presentation data structures 506. It should be noted that the loudness data 510 typically comprises several loudness data instances, one for each of the one or more presentation data structures 506.

[0126] The encoder 500 may also be adapted to determine, for each of the one or more presentation data structures 506, dynamic range compression DRC data for the referenced one or more content substreams. The DRC data quantifies at least one desired compression curve or at least one set of DRC gains. The DRC data is included in the bitstream P. The DRC data and the loudness data 510 may be included in a metadata substream 614 according to an embodiment. As discussed above, the loudness data is typically presentation-dependent. Moreover, the DRC data may also be presentation-dependent. In this case, the loudness data for a particular presentation data structure, and if applicable, the DRC data as well, are included in a dedicated metadata substream 614 for that particular presentation data structure.

[0127] The encoder may also be adapted to apply, for each of the plurality of content substreams 502, a predefined loudness function to obtain substream-level loudness data for the content substream; and include the substream-level loudness data in the bitstream. The predefined loudness function may be related to a gating of the audio signal. According to other embodiments, the predefined loudness function is only related to such time periods of the audio signal representing dialogues. The predefined loudness function may, according to some embodiments, comprise at least one of the following:

[0128] Frequency-dependent weighting of audio signals;

[0129] Channel-dependent weighting of the audio signal;

[0130] Ignore segments of the audio signal where the signal power is below a threshold;

[0131] ● Ignore segments of the audio signal that are detected as not being speech;

[0132] Calculate energy / power / RMS measurements of audio signals.

[0133] As understood from the above, the loudness function is non-linear. This means that, in case the loudness data is calculated only from different content substreams, the loudness for a certain presentation cannot be calculated by adding together the loudness data of the referenced content substreams. Moreover, when different audio tracks (i.e., content substreams) are combined together for simultaneous playback, combining effects between coherent / incoherent parts or in different frequency regions of different tracks may occur, which further makes the addition of loudness data for the tracks mathematically impossible.

[0134] IV. Equivalence, extension, substitution and others

[0135] After studying the above description, further embodiments of the present disclosure will become clear to those skilled in the art. Even though the present description and the accompanying drawings disclose embodiments and examples, the present disclosure is not limited to these specific examples. Many modifications and variations may be made without departing from the scope of the present disclosure, which is defined by the appended claims. Any reference numerals appearing in the claims are not to be construed as limiting their scope.

[0136] In addition, variations to the disclosed embodiments may be understood and implemented by those skilled in the art who practice the present disclosure by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage.

[0137] The devices and methods disclosed above can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementation, the division of tasks between the functional units cited in the above description does not necessarily correspond to the division into physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in cooperation. Some or all components can be implemented as software executed by a digital signal processor or a microprocessor, or implemented as hardware or implemented as a dedicated integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media for storing information (such as computer-readable instructions, data structures, program modules or other data) implemented by any method or technology. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modular data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

Claims

1. A method comprising: Obtaining the encoded bit stream by a decoding device; extracting, by the decoding device, an audio signal and metadata from the encoded bitstream, the metadata comprising dynamic range compression (DRC) data and loudness data, wherein the DRC data indicates a compression curve corresponding to one of a plurality of DRC profiles; generating, by the decoding device, one or more loudness values ​​using the loudness data; mapping, by the decoding device, the one or more loudness values ​​to DRC gains using the compression curve; and The DRC gain is applied to the audio signal by the decoding device.

2. The method according to claim 1, wherein: The DRC gains are applied to groups of channels.

3. The method according to claim 2, wherein: At least some of the loudness data is associated with particular channels in the plurality of groups of channels.

4. The method according to claim 1, wherein: The loudness data comprises a loudness function comprising a channel-dependent weighting of the audio signal.

5. The method according to claim 1, wherein: Mapping the loudness values ​​to the DRC gains includes ignoring segments of the audio signal that are not detected as speech.

6. A decoding device, comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: Obtaining an encoded bit stream; extracting an audio signal and metadata from the encoded bitstream, the metadata comprising dynamic range compression (DRC) data and loudness data, wherein the DRC data indicates a compression curve corresponding to one of a plurality of DRC profiles; generating one or more loudness values ​​using the loudness data; mapping the one or more loudness values ​​to DRC gains using the compression curve; and The DRC gain is applied to the audio signal.

7. The decoding device according to claim 6, wherein: The DRC gains are applied to groups of channels.

8. The decoding device according to claim 7, wherein: At least some of the loudness data is associated with particular channels in the plurality of groups of channels.

9. The decoding device according to claim 6, wherein: The loudness data comprises a loudness function comprising a channel-dependent weighting of the audio signal.

10. The decoding device according to claim 6, wherein: Mapping the loudness values ​​to the DRC gains includes ignoring segments of the audio signal that are not detected as speech.

11. A non-transitory computer-readable storage medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform the method according to any one of claims 1-5.

12. A computer program product comprising executable instructions for performing the method according to any one of claims 1 to 5 when executed on a computer.

13. A decoder for processing a bitstream (P) comprising a plurality of content substreams (412), each content substream representing an audio signal, the decoder comprising: A receiving component configured to receive the bit stream; a demultiplexer (102) configured to extract from said bitstream (P) one or more presentation data structures (104), each presentation data structure comprising references (404, 405) to a plurality of said content substreams and further comprising a reference (406) to loudness data (408) comprised in the metadata substream (205), wherein said loudness data is specific to said presentation data structure and indicates what loudness a combination of the referenced plurality of content substreams (204) will have when decoded; a playback state component (106) configured to receive data (108) indicating a selected presentation data structure (110) among the one or more presentation data structures (104) and a desired loudness level; as well as a mixing component (112) configured to decode the plurality of content substreams (204) referenced by the selected presentation data structure (110) and to form an output audio signal (114) based on the decoded content substreams (204), The mixing component (112) is further configured to process the decoded plurality of content substreams or output audio signals to achieve the desired loudness level based on the loudness data referenced by the selected presentation data structure (110).

14. An audio encoder (500), comprising: a loudness component (508) configured to apply a predefined loudness function (514) to obtain loudness data (510) indicating what loudness a combination of the plurality of content substreams representing the respective audio signals will be when decoded by a decoder; a presentation data component (504) configured to define one or more presentation data structures (506), each presentation data structure comprising references (604, 605) to a plurality of content substreams (612) among the plurality of content substreams (502) and a reference (608) to loudness data (510) of what loudness a combination of the referenced content substreams will be when decoded by a decoder; as well as A multiplexing component (512) is configured to form a bitstream (P), the bitstream (P) comprising the plurality of content substreams, the one or more presentation data structures (506) and the loudness data (510) referenced by the one or more presentation data structures.

Images