Adaptive loudness normalization for audio object clustering

By clustering audio elements and applying compensation gains based on energy and correlation measures, the method addresses volume boosting issues in audio processing, improving audio quality across different rendering scenarios.

JP7596281B2Active Publication Date: 2024-12-09DOLBY LABORATORIES LICENSING CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021547121
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-11
Filing Date
2020-02-12
Publication Date
2024-12-09
Estimated Expiration
2040-02-12

AI Technical Summary

Technical Problem

Existing audio processing methods for multiple audio elements, such as audio objects and beds, suffer from volume boosting issues when clustering and rendering to speaker layouts, particularly in bandwidth-limited environments, leading to undesirable audio quality.

Method used

A method and apparatus for processing audio content that involves clustering audio elements into groups, determining energy and correlation measures for each element, and applying compensation gains to mitigate volume differences, ensuring consistent volume levels across different rendering scenarios.

Benefits of technology

The method significantly reduces volume boosting and enhances perceived audio quality by ensuring consistent volume levels across various speaker layouts and environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596281000060
    Figure 0007596281000060
  • Figure 0007596281000061
    Figure 0007596281000061
  • Figure 0007596281000062
    Figure 0007596281000062
Patent Text Reader

Abstract

1. A method for processing audio content comprising a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; for a cluster among the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; for at least one audio element in the cluster, determining a compensation gain based at least in part on the energy metric of the audio element in the cluster; and applying the compensation gain to the at least one audio element in the cluster.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Related Applications] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 814,718, filed March 6, 2019, and European Patent Application No. 19161889.1, filed March 11, 2019, and PCT / CN2019 / 074915, filed February 13, 2019, which applications are incorporated herein by reference in their entireties. [Technical field] The present disclosure relates to methods and apparatus for processing audio content that includes multiple audio elements, and in particular to adaptive loudness normalization of such audio content. [Background technology]

[0002] The new consumer Dolby® Atmos® Cinema system introduces a new audio format that includes audio beds (channels) and audio objects. An audio bed represents an audio channel that is meant to be played at a given fixed speaker position. An audio object, on the other hand, represents a separate audio element that exists for a defined period of time and may also have spatial information (e.g., as part of the metadata) that describes the position, speed, and size of each object. During transmission, the bed and objects are transmitted separately and can be used by a spatial reproduction system that recreates the artistic intent with a variable number of speakers at known physical positions. In some soundtracks, there may be up to 7, 9, or 11 bed channels. Furthermore, based on the capabilities of the authoring system, there may be tens or hundreds of separate audio objects that are combined during rendering to create spatial diversity and an immersive audio experience.

[0003] The large number of audio signals present in such object-based content presents new challenges for the coding and distribution of such content. In some distribution and transmission systems, there may be enough available bandwidth to transmit the entire audio bed and objects with little or no audio compression. However, in some cases, such as Blu-ray Disc, broadcast (cable, satellite, and terrestrial), mobile (3G and 4G), and over the top (OTT, or Internet) distribution, there may be significant limitations in the available bandwidth to digitally transmit the entire bed and objects. Although audio coding methods (lossy or lossless) can be applied to the audio to reduce the required bandwidth, audio coding may not be sufficient to reduce the bandwidth required to transmit audio, especially over very limited networks such as mobile 3G and 4G networks.

[0004] To solve this problem, the number of input objects and beds can be reduced to a smaller set of output objects / beds by clustering. In general, the audio clustering process includes two main stages: (1) determining the cluster locations and (2) determining the gains of rendering objects into output clusters, with the aim of minimizing the overall spatial distortion or preserving the overall spatial perception based on the spatial mask assumption.

[0005] Clustering may usually work well when objects / beds are clustered into a significant number of clusters (e.g., 11). However, this is usually not the case for the "cascaded audio object clustering" use case. This use case is shown diagrammatically in Fig. 1. Object-based audio content 110 (e.g., Atmos print masters) is clustered into a first number (e.g., 11) of (intermediate or initial) clusters in a first clustering stage 120. The obtained clusters are then further clustered into a smaller number (final or output) clusters (e.g., 5) in a second clustering stage 130. In this use case, when the final clusters (e.g., 5) are rendered in a given speaker layout (e.g., 5.1.2) in the processing stage 140, a loudness boost is seen compared to directly rendering the initial clusters (e.g., 11) into the same speaker layout. This loudness boost is clearly undesirable.

[0006] A similar (less noticeable) loudness boost may occur in a use case where the objects / bed are directly clustered into a cluster number (e.g. 5) and then rendered into a speaker layout. This use case is shown diagrammatically in Figure 2. The object-based audio content 210 is clustered into a cluster number (e.g. 5) in a clustering stage 220 and then rendered into a speaker layout in a processing stage 230.

[0007] There is therefore a need for improved handling of audio content that includes multiple audio elements. In particular, there is a need for improved handling of audio content that includes multiple audio elements that avoids volume boosting when rendering a clustered version of the audio content to a speaker layout. In general, there is a need for improved control of the volume of such audio content. Summary of the Invention

[0008] The invention provides a method for processing audio content comprising a plurality of audio elements and a corresponding device, each having the features of the independent claims.

[0009] An aspect of the disclosure relates to a method for processing audio content comprising a plurality of audio elements. The audio elements may be localized audio elements, for example comprising audio objects, audio beds (bed channels), and / or (intermediate) clusters of audio objects. The method may comprise clustering the plurality of audio elements into a plurality of clusters of audio elements (e.g. final clusters or output clusters). Each of the clusters may comprise spatially close audio elements. The number of clusters may be less than the number of audio elements. Processing may be applied to each cluster. Thus, the method comprises, for a cluster among the plurality of clusters: The method may further comprise the step of: for each audio element in the cluster, determining an energy metric that the audio element contributes to the cluster. The method may further include determining, for at least one audio element in the cluster, a compensation gain based at least in part on the energy measure of the audio element in the cluster. The method may further include, for a cluster among the plurality of clusters, applying a compensation gain to at least one audio element in the cluster. Applying the compensation gain to the at least one audio element may reduce a difference in loudness between the at least one audio object when rendered onto a set of speakers (layout) as part of a cluster and the at least one audio object when rendered directly onto the set of speakers. The method may further include rendering a plurality of clusters of audio elements onto a speaker layout.

[0010] The step of determining the compensation gains in the proposed method can significantly reduce the loudness boost, i.e. the loudness of each perceptible audio object or bed channel resulting from rendering the clusters into the target speaker layout can be much closer to the respective loudness that would result if the audio object or bed channel were directly rendered into the target speaker layout.

[0011] In some embodiments, a measure of the energy that an audio element contributes to a cluster c may be given by:

number

[0012] In some embodiments, the method may further include determining, for a cluster among the plurality of clusters, a spectrum of the cluster based on respective spectra that the audio elements contribute to the cluster. The method may further comprise determining, for each audio element in the cluster, as at least part of the compensation gain, an overall compensation gain for the cluster based at least in part on the energy measures of the audio elements in the cluster and the spectrum of the cluster.

[0013] In some embodiments, the method may further include determining, for a cluster among the plurality of clusters, a first energy index of the cluster as a sum of energy indexes that audio elements in the cluster contribute to the cluster. The method may further include determining, for a cluster among the plurality of clusters, a spectrum of the cluster based on respective spectra that audio elements contribute to the cluster. The method may further include determining, for a cluster among the plurality of clusters, a second energy index of the cluster based on the spectrum of the cluster. The first energy index may be referred to as a total energy of the cluster (total element energy (e.g., total object energy) or expected energy). The second energy index may be referred to as an actual energy of the cluster. The method may further include determining, for a cluster among the plurality of clusters, for each audio element in the cluster, an overall compensation gain of the cluster based on the first energy index and the second energy index as at least a portion of the compensation gain.

[0014] Applying an overall compensation gain to the audio elements in a cluster reduces the difference between the estimated and actual energy of the cluster, thereby reducing the volume boost and improving the perceived audio quality.

[0015] In some embodiments, the first energy index of a cluster is given by:

number

number

number

[0016] In some embodiments, the overall compensation gain of the cluster is determined as the square root of the ratio of the first energy index and the second energy index. For example, the overall compensation gain of the cluster is given by:

number

number

[0017] In some embodiments, the method may further include, for a given audio element in a cluster among the plurality of clusters, determining a correlation metric between the given audio element and any of the plurality of audio elements. The method may further include determining, as at least part of the compensation gain for a given audio element, an individual compensation gain for the given audio element based at least in part on the energy indicators of the audio elements in the cluster and the correlation indicator between the given audio element and any of the plurality of audio elements.

[0018] In some embodiments, the method may further include, for a given audio element in a cluster among the plurality of clusters, determining a correlation index between the given audio element and any of the plurality of audio elements. The method may further include, for a cluster among the plurality of clusters, determining a third energy index for the given audio element as a sum of energy indexes that the audio elements contribute to the cluster. The weight of the energy index may be based on the respective correlation index between the respective audio elements and the given audio element. The method may further include, for a given audio element in a cluster among the plurality of clusters, determining a fourth energy index for the given audio element as a weighted sum of a geometric mean of the energy indexes that the given audio element contributes to the cluster and the respective energy indexes that the audio elements in the plurality of audio elements other than the given audio element contribute to the cluster, over any audio elements among the plurality of audio elements except the given audio element. The weight of the geometric mean may be based on the respective correlation indexes between the respective audio elements and the given audio element. The method may further include determining, for a given audio element in a cluster among the plurality of clusters, an individual compensation gain for the given audio element based on the third energy index and the fourth energy index as at least a portion of the compensation gain for the given audio element.

[0019] The step of applying individual compensation gains to audio elements in a cluster attenuates audio elements depending on their correlation with other audio elements. The overall idea is as follows: if an audio element is highly correlated with other audio elements, it will introduce a higher volume boost and therefore the application of a smaller gain will be more appropriate. This allows a targeted attenuation of audio elements, since the more highly correlated an audio element is, the stronger it contributes to the volume boost, thereby reducing the volume boost and improving the perceived speech quality.

[0020] In some embodiments, the correlation measure between a given audio element and any of the multiple audio elements may be given by:

number

number

number

number

[0021] In some embodiments, the method may further include determining, for a cluster among the plurality of clusters, a respective individual compensation gain for each audio element in the cluster. The method may further include, for a cluster among the plurality of clusters, applying the respective individual compensation gains to the audio elements in the cluster to obtain an individual compensation audio element. The method may further include, for a cluster among the plurality of clusters, determining a spectrum of the cluster based on respective spectra that the individual compensation audio elements contribute to the cluster. The method may further include, for a cluster among the plurality of clusters, determining: The method may further include determining, for each individual compensation audio element in the cluster, an overall compensation gain for the cluster based at least in part on the energy measures of the individual compensation audio elements in the cluster and the spectrum of the cluster as at least a portion of the compensation gain.

[0022] In some embodiments, the method may include determining, for a cluster among the plurality of clusters, a respective individual compensation gain for each audio element in the cluster. The method may further include, for a cluster among the plurality of clusters, applying a respective individual compensation gain to the audio elements in the cluster to obtain an individual compensation audio element. The method may further include determining, for a cluster among the plurality of clusters, a fifth energy index of the cluster as a sum of energy indices that the individual compensation audio elements in the cluster contribute to the cluster. The method may further include, for a cluster among the plurality of clusters, determining a spectrum of the cluster based on a respective spectrum that the individual compensation audio elements contribute to the cluster. The method may further include determining, for a cluster among the plurality of clusters, a sixth energy index of the cluster based on the spectrum of the cluster. Thus, the fifth energy index may correspond to the first energy index and the sixth energy index may correspond to the second energy index, the difference being where the individual compensation audio element is taken into account. The method may further include, for a cluster among the plurality of clusters, determining a fifth energy index of the cluster as a sum of energy indices that the individual compensation audio elements in the cluster contribute to the cluster. The method may further include determining, for each individual compensation audio element in the cluster, an overall compensation gain for the cluster based on the fifth energy index and the sixth energy index (e.g., as the square root of their ratio, in the same way as for the first and second energy indexes) as at least part of the compensation gain.

[0023] By determining such an overall compensation gain after the individual compensation gains have been applied, the loudness boost is further reduced and the perceived sound quality is further improved.

[0024] In some embodiments, the method may further include determining, for a speaker on which at least one of the clusters is rendered, a respective energy metric that the audio elements contribute to an output (e.g., an output signal) of the speaker. The method may further include determining, for a speaker on which at least one of the clusters is rendered, a spectrum of the speaker's output based on the respective spectra that the audio elements contribute to the speaker's output. The method may further include determining, for a speaker on which at least one of the clusters is rendered, an overall compensation gain of the speaker based at least in part on the energy metric that the audio elements contribute to the speaker's output and the spectrum of the speaker's output.

[0025] In some embodiments, the method may further include determining, for a speaker on which at least one of the clusters is rendered, a respective energy metric that the audio elements contribute to the output of the speaker (e.g., an output signal). The audio elements may be original audio elements or individual compensation audio elements. The method may further include determining, for a speaker on which at least one of the clusters is rendered, a seventh energy metric of the speaker's output based on the respective energy metric that the audio elements contribute to the output of the speaker. The method may further include determining, for a speaker on which at least one of the clusters is rendered, a spectrum of the speaker's output based on the respective spectrum that the audio elements contribute to the output of the speaker. The method may further include determining, for a speaker on which at least one of the clusters is rendered, an eighth energy metric of the speaker's output based on the spectrum of the speaker's output. The method may further include determining, for a speaker on which at least one of the clusters is rendered, an overall compensation gain of the speaker based on the seventh energy metric and the eighth energy metric.

[0026] By determining such speaker-dependent compensation gains (possibly after global and / or individual compensation gains have been applied), the loudness boost can be further reduced and the perceived sound quality further improved.

[0027] In some embodiments, the seventh energy index may be given by the following formula:

number

number

number

[0028] In some embodiments, the overall compensation gain of the speaker may be determined as the square root of the ratio of the seventh energy index and the eighth energy index. For example, the overall compensation gain of the speaker may be given by the following formula:

number

[0029] In some embodiments, clustering the audio elements into clusters may include clustering the audio elements into intermediate clusters (stage 1 clustering). Clustering the audio elements into clusters may further include clustering the intermediate clusters into clusters (stage 2 clustering). This clustering may be referred to as cascaded audio object clustering.

[0030] In some embodiments, the method may further comprise the step of applying a dynamic range compressor or limiter to the determined compensation gains before applying the compensation gains to the respective audio elements.

[0031] In some embodiments, the method may further include setting the compensation gain to unity depending on whether the difference between the expected (e.g., total) energy and the actual energy of each cluster is less than a predetermined threshold of difference. For example, if the difference is less than a predetermined threshold, the compensation gain may be set to unity (i.e., no additional compensation).

[0032] In some embodiments, the method may further comprise increasing the decorrelation between audio elements among the plurality of audio elements having a spatial size above a predetermined threshold for size, the additional decorrelation being applied in particular to the inner bed channel.

[0033] In some embodiments, a compensation gain may be determined in each of a number of frequency subbands.

[0034] In some embodiments, the energy measure may be a loudness measure, i.e. the compensation gain determination may be performed in the loudness domain.

[0035] These indices allow further refinement of the compensation gain determination.

[0036] Another aspect of the present disclosure relates to an apparatus including a processor and a memory coupled to the processor storing instructions for execution by the processor, the processor may be configured to perform the method according to the aforementioned aspects and any of the method steps of the embodiments thereof.

[0037] Another aspect of the present disclosure relates to a computer program comprising instructions that, when executed by a processor, cause the processor to perform the method according to the first aspect above and any of its embodiments.

[0038] Another aspect of the present disclosure relates to a computer readable storage medium storing a computer program according to the previous aspect.

[0039] Although this disclosure refers to audio elements within a given cluster, it is understood that a given audio element can be rendered into more than one cluster according to the respective element-cluster gains. In this sense, the audio elements within a given cluster can be understood to be the portion of the audio element that is rendered into the given cluster. Providing a particular compensation gain to a portion of an audio element does not exclude a different compensation gain being applied to another portion of the audio element. [Brief description of the drawings]

[0040] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numbers indicate like or similar elements, and in which: [Figure 1] 1 illustrates a schematic diagram of a first example of use of an embodiment of the present disclosure. [Diagram 2] 2 illustrates a second example of use of an embodiment of the present disclosure; [Diagram 3] 1 is a flowchart illustrating an example method for processing audio content according to an embodiment of the present disclosure. [Figure 4] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Diagram 5] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 6] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 7] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 8] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 9] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 10] 4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. [Figure 11]4 is a flow chart illustrating an example embodiment of the method of FIG. 3 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] As mentioned above, in this disclosure, the same or similar reference numerals indicate the same or similar elements, and repeated descriptions thereof may be omitted for reasons of brevity.

[0042] As can be seen, the loudness boost is mainly caused by objects with sizes (possibly zone masks) that are first pre-baked to the internal speaker layout (e.g. 7.1.4) before clustering into clusters. When these internal beds are grouped into dynamic clusters, or the clusters obtained from the first stage clustering process are further grouped into fewer clusters in the second stage, signals from the same object distributed into different beds or clusters are further rendered into the same cluster and acoustically summed in the subsequent clustering process, thus introducing a loudness boost.

[0043] Usually, loudness boosting may be content-dependent, cluster-dependent, and speaker layout-dependent. Therefore, it is not practical to use a predefined gain for each object / cluster to compensate for the loudness boosting. This disclosure presents an adaptive loudness normalization method to solve this problem.

[0044] As mentioned above, the process according to the embodiment of the present disclosure is applicable to at least two use cases: cascaded clustering of content based on objects followed by rendering to a speaker layout (first use case); and direct rendering of clustered audio content to a speaker layout (second use case, especially when there is a limited number of clusters). To solve these use cases together, the term "audio element" is used throughout the present disclosure to mean a localized audio element, such as an audio object, an audio bed (bed channel), and / or an (intermediate) cluster of audio objects or audio beds. Furthermore, unless otherwise stated, a cluster means a cluster intended for rendering. Clusters that themselves undergo further clustering may be referred to as audio elements or intermediate clusters. Using this terminology, it can be said that cascaded clustering relates to clustering multiple audio elements by first clustering multiple audio elements into multiple intermediate clusters and then clustering multiple intermediate clusters into multiple clusters.

[0045] Broadly speaking, processing according to embodiments of the present disclosure includes the steps of analyzing the expected energy and the actual energy of each cluster, calculating a corresponding compensation gain g, and then multiplying any original element-cluster gains (e.g., object-cluster gains) g for each audio element (e.g., audio object, audio bed, or middle cluster) o in a given cluster c. oc and applying the calculated gain.

[0046] Depending on different use cases, not all audio elements need compensation gain. In line with the above discussion, in some embodiments, compensation gain may be applied to intermediate clusters in cascade clustering (first use case, FIG. 1) and to inner beds with a given (pre-baked) object size in the single stage case (second use case, FIG. 2). However, the application field of the embodiments of the present disclosure is not limited to these examples and compensation gain may be applied to other entities as well.

[0047] A first example of a method 300 for processing audio content comprising multiple audio elements is shown in figure 3. Again, the audio elements may be related to audio objects or audio beds (e.g. in the second use case) or to (intermediate) clusters of audio objects or audio beds (e.g. in the first use case).

[0048] In step S310, the multiple audio elements are clustered into multiple clusters of audio elements, where each cluster may include audio elements that are spatially close, and the number of clusters may be less than the number of audio elements.

[0049] Steps S320-S340 are subsequently performed for (at least) one of the plurality of clusters, and of course, in some embodiments, the process may be applied to each of the plurality of clusters.

[0050] In step S320, for each audio element in the cluster, a measure of the energy that the audio element contributes to the cluster is determined (e.g., calculated). For example, the energy measure E oc is given by:

number

[0051] In step S330, a compensating gain is determined (eg, calculated) for at least one audio element in the cluster based at least in part on the energy measure of the audio element in the cluster.

[0052] In step S340, a compensation gain is applied to at least one audio element in the cluster. Applying a compensation gain to the at least one audio element may reduce a difference in volume between the at least one audio object when rendered to a set of speakers as part of a cluster and the at least one audio object when rendered directly to the set of speakers.

[0053] In some embodiments, the method 300 may further include rendering multiple clusters of audio elements into a speaker layout.

[0054] 4-11, more specific implementation examples and details of method 300 are described. As will become clear from these examples, the compensation gain (e.g., determined in step S330) may include any of a global compensation gain for a given cluster (which is the same for all audio elements in a given cluster), an individual compensation gain (which may vary between audio elements in a given cluster), and / or a global compensation gain for a speaker (which is the same for all audio elements rendered to a given speaker). Any of the methods described below can be viewed as an implementation of step S330 of method 300.

[0055] 4 and 5 show methods 400 and 500, respectively, that return (and apply) a global compensation gain per cluster, i.e., they can be said to be related to cluster-adaptive loudness normalization.

[0056] The general idea underlying these methods is to estimate an adaptive gain for each audio element (e.g. object) in a cluster when the audio element is rendered into the cluster (the gain is uniform across the cluster). For each cluster, the total energy (total element energy (e.g. total object energy) or expected energy) contributed to the cluster by all objects rendered into the cluster is calculated. Then the actual energy of the cluster is calculated, and finally a compensating gain is calculated to reduce the difference between the total energy and the actual energy.

[0057] Method 400 of Figure 4 can be seen to be a high-level implementation of this general idea. Steps S410 and S420 are performed for the aforementioned clusters in the plurality of clusters. In some embodiments, they may be performed for each cluster in the plurality of clusters.

[0058] In step S410, the spectrum of the cluster is determined (eg, calculated) based on the spectra of each of the audio elements that contribute to the cluster.

[0059] In step S420, for each audio element in the cluster, an overall compensation gain for the cluster is determined (eg, calculated) based at least in part on the energy measures of the audio elements in the cluster and the spectrum of the cluster as the compensation gain.

[0060] 5 is a particular implementation of method 400. Steps S510-S540 are performed for the aforementioned clusters among the plurality of clusters. In some embodiments, they may be performed for each cluster among the plurality of clusters.

[0061] In step S510, a first energy index for the cluster is determined (e.g., calculated) as the sum of the energy indices that the audio elements in the cluster contribute to the cluster. The first energy index is the total energy of the cluster, E tot_o, i.e. the total (object) energy rendered in cluster c. Then the first energy index for cluster c is given by:

number

[0062] In step S520, a spectrum of the cluster is determined (e.g., calculated) based on the spectra of each of the audio elements that contribute to the cluster. c may be given by the formula:

number

[0063] In step S530, a second energy index of the cluster is determined based on the spectrum of the cluster. The second energy index is the actual energy E c The second energy measure may then be given by:

number

number

number

[0064] Here, and in the remainder of this disclosure, the compensation gains may be determined on a frame-by-frame (dynamic) basis, i.e., the compensation gains may be determined for each frame or group of frames of the audio content. Furthermore, smoothing may be applied to the compensation gains determined on a frame-by-frame (or group-by) basis.

[0065] 6 and 7 show methods 600 and 700, respectively, that return (and apply) correlation-dependent compensation gains, i.e., they can be said to be related to correlation-dependent factor adaptive loudness normalization.

[0066] Methods 400 and 500 estimate one gain per cluster and apply the same gain for all audio elements rendered in that cluster. In contrast, methods 600 and 700 determine element-adaptive (e.g. object-adaptive) gains and apply different gains to different audio elements. For this purpose, correlation between audio elements is exploited. The overall idea is as follows: if an audio element is highly correlated with other audio elements, it will introduce a higher volume boost and therefore applying a smaller gain will be more appropriate.

[0067] Method 600 of Figure 6 can be seen to be a high-level implementation of this general idea. Steps S610 and S620 are performed for a given audio element in said cluster among the plurality of clusters. In some embodiments, they may be performed for each audio element in the cluster and / or for each cluster among the plurality of clusters.

[0068] In step S610, a correlation measure between a given audio element and any of a number of audio elements (typically, although not necessarily within the same cluster) is determined (eg, calculated).

[0069] In step S620, an individual compensation gain for the given audio element is determined (e.g., calculated) based at least in part on the energy indicators of the audio elements in the cluster and the correlation indicators between the given audio element and any of the multiple audio elements as the compensation gain for the given audio element.

[0070] 7 is a particular implementation of method 600. Steps S710-S740 are performed for a given audio element in said cluster among the plurality of clusters. In some embodiments, they may be performed for each audio element in the cluster and / or for each cluster among the plurality of clusters.

[0071] In step S710, a correlation index between the given audio element and any of the plurality of audio elements is determined (e.g., calculated). A correlation index r between the given audio element o and any of the plurality of audio elements U is determined (e.g., calculated). ou is given by:

number

[0072] In step S720, an energy measure E of the contribution of audio element u to cluster c is calculated. uc A third energy measure for the given audio element may be determined (e.g., calculated) as a weighted sum of a, b, c, d, e ... oc may be given by the following formula:

number

number

number

[0073] In step S730, a fourth energy index for the given audio element is determined (e.g., calculated) as a weighted sum of the geometric mean of the energy index of the given audio element's contribution to the cluster and the respective energy index of the audio elements in the plurality of audio elements other than the given audio element, over any audio elements in the plurality of audio elements other than the given audio element, where the weight of the geometric mean may be based on the respective correlation index between each audio element and the given audio element. For example, the fourth energy index b oc may be given by the following formula:

number

[0074] In step S740, for each given audio element, an individual compensation gain for the given audio element is determined (e.g., calculated) based on at least the third and fourth energy indices as a compensation gain. For example, the individual compensation gain g1 oc may be given by the following formula:

number

[0075] For example, if the correlation matrix is ​​for three audio elements (e.g., objects):

number

[0076] Alternatively, each audio element o in cluster c may be assigned an individual compensation gain g oc After applying the compensation energy E o and Spectrum X c (i.e., the energy and spectrum after application of the individual compensation gains), a global compensation gain g1 is applied to cluster c so as to minimize the difference between the expected energy and the actual energy of cluster c. c can be determined (e.g., calculated). For cluster c, the individual compensation gain g oc Determine the individual compensation gain g1 oc Applying the overall compensation gain g1 cFor each audio element o in cluster c, a compensation gain g1' is determined by oc can be determined:

number

number

[0077] The method 800 of FIG. 8 calculates the overall gain g1′ oc It can be seen that this is a high-level implementation of the determination of the number of clusters. Steps S810 to S840 are performed for the aforementioned clusters among the plurality of clusters. In some embodiments, they may be performed for each cluster among the plurality of clusters.

[0078] In step S810, for each audio element in the cluster, a respective individual compensation gain is determined (e.g., calculated). This may proceed, for example, by method 600 or 700.

[0079] In step S820, each individual compensation gain is applied to the audio elements in the cluster to obtain an individual compensation audio element.

[0080] In step S830, the spectrum of the cluster is determined (eg, calculated) based on the spectra that each of the individual compensation audio elements contributes to the cluster.

[0081] In step S840, for each individual compensation audio element in the cluster, an overall compensation gain for the cluster is determined (e.g., calculated) based at least in part on the energy indicators of the individual compensation audio elements in the cluster and the spectrum of the cluster as the compensation gain.

[0082] In general, method 800 can be said to correspond to successively performing methods 400 / 500 after individual compensation gains have been applied per method 600 / 700 to the audio elements in the cluster.

[0083] 9 is a specific implementation of method 800. Steps S910-S960 are performed for a cluster among the plurality of clusters. In some embodiments, they may be performed for each cluster among the plurality of clusters.

[0084] In step S910, for each audio element in the cluster, a respective individual compensation gain is determined (e.g., calculated). This may proceed, for example, according to method 600 or 700.

[0085] In step S920, each individual compensation gain is applied to the audio elements in the cluster to obtain an individual compensation audio element.

[0086] In step S930, a fifth energy index for the cluster is determined (e.g., calculated) as the sum of the energy indexes that the individual compensated audio elements in the cluster contribute to the cluster. The fifth energy index may correspond to the first energy index described above, except that the individual compensated audio elements (instead of the initial uncompensated audio element) are taken into account. This may therefore proceed similarly to step S510 described above.

[0087] In step S940, the spectrum of the cluster is determined (eg, calculated) based on the spectra that each of the individual compensation audio elements contributes to the cluster, which may proceed similarly to step S520 described above.

[0088] In step S950, a sixth energy index of the cluster is determined (e.g., calculated) based on the spectrum of the cluster. The sixth energy index may correspond to the second energy index, with the difference that the individual compensated audio elements (instead of the initial uncompensated audio elements) are taken into account. This may therefore proceed similarly to step S530 described above.

[0089] Finally, in step S960, an overall compensation gain for the cluster is determined (e.g., calculated) based on the fifth and sixth energy indices as at least a part of the compensation gain for each individual compensation audio element in the cluster, which may proceed similarly to step S540 described above.

[0090] 10 and 11 show methods 1000 and 1100, respectively, which return (and apply) a global compensation gain for each speaker of the (target) speaker layout in which the clusters are rendered, i.e., they are said to be related to speaker-adaptive loudness normalization. The resulting speaker-adaptive gains can be applied in addition to the gains determined by methods 400-900 above.

[0091] The general idea is that if the playback speaker layout is known, the target speaker layout can be used to estimate the appropriate gains to further minimize the possible volume boost.

[0092] It can be seen that the method 1000 of FIG. 10 is a high-level implementation of determining speaker-specific overall compensation gains. Steps S1010-S1030 are performed for a speaker for which at least one of the multiple clusters is rendered. In some embodiments, they may be performed for each speaker for which at least one of the multiple clusters is rendered. The audio elements in this method may be original / initial audio elements or audio elements compensated by any of the aforementioned compensation gains (e.g., individual compensation audio elements, etc.).

[0093] In step S1010, an energy metric for each audio element that contributes to the output of a speaker (eg, an output signal, a speaker channel signal) is determined (eg, calculated).

[0094] In step S1020, the spectrum of the speaker's output is determined (eg, calculated) based on the respective spectra that the audio elements contribute to the speaker's output.

[0095] In step S1030, an overall compensation gain for the speaker is determined (eg, calculated) based at least in part on the energy measure that the audio element contributes to the output of the speaker and the spectrum of the output of the speaker.

[0096] Method 1100 of Figure 11 is a specific implementation of method 1000. The method includes calculating the total elemental energy (e.g., object energy) rendered to a given speaker channel and calculating the actual spectrum and actual energy of the signal received / formed by the speaker channel. Thus, speaker-dependent compensation gains can be calculated in response.

[0097] Steps S1110-S1150 are performed for a speaker for which at least one of the plurality of clusters is rendered. In some embodiments, they may be performed for each speaker for which at least one of the plurality of clusters is rendered. The audio elements in this method may be original / initial audio elements or audio elements compensated by any of the aforementioned compensation gains (e.g., individual compensation audio elements, etc.).

[0098] In step S1110, an energy metric for each audio element that contributes to the output of a speaker (eg, an output signal, a speaker channel signal) is determined (eg, calculated).

[0099] In step S1120, a seventh energy metric for the speaker's output is determined (e.g., calculated) based on the respective energy metrics that the audio elements contribute to the speaker's output. The seventh energy metric may be referred to as the overall element energy (e.g., object energy) expected to be rendered by speaker (speaker channel) s. For example, the seventh energy metric may be given by the following formula:

number

[0100] In step S1130, the spectrum of the speaker's output is determined (e.g., calculated) based on the respective spectra that the audio elements contribute to the speaker's output. cls→spk may be expressed as the actual signal received by speaker (speaker channel) s, which is given by:

number

[0101] In step S1140, an eighth energy measure of the speaker's output is determined (e.g., calculated) based on the spectrum of the speaker's output. The eighth energy measure may be referred to as the (actual) energy in the speaker (speaker channel). It is given by the following formula:

number

[0102] In step S1150, an overall compensation gain for the speaker is determined (e.g., calculated) based on the seventh energy index and the eighth energy index. The overall compensation gain for the speaker may be determined as the square root of the ratio of the seventh energy index and the eighth energy index. For example, the overall compensation gain for the speaker g2 oc is given by:

number

[0103] As mentioned above, the overall compensation gain g oc can be combined with any of the compensation gains obtained by methods 400 / 500, 600 / 700, or 800 / 900 and applied in addition to the original element-cluster gains, i.e., the resulting element-cluster gains are given by:

number

[0104] To make any of the above compensation gains more stable and undisturbed, a compressor (e.g., a dynamic range compressor, a limiter) may be applied to the obtained compensation gain. For example, the minimum and maximum values ​​of the compensation gain may be limited. Thus, a method according to an embodiment of the present disclosure (e.g., method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100) may include applying a dynamic range compressor or limiter to the determined compensation gain before applying the compensation gain to the respective audio element. For example, the gain value may be limited to the range (0.25, 4), i.e., [-6 dB, 6 dB] in the decibel domain.

[0105] In some embodiments, a relax parameter can be added: if the difference between the expected energy (first or fifth energy index) and the actual energy of a cluster (second or sixth energy index) is less than a tolerance threshold, e.g. 1 dB, the difference can be accepted and the global compensation gain of the cluster can be set to 1 (unity). In this case, the global compensation gain of the cluster is only applied when the difference is large.

[0106] Typically, a method according to an embodiment of the present disclosure (e.g., method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100) may further include setting the compensation gain to unity depending on whether the difference between the expected energy and the actual energy of the respective cluster is less than a predetermined threshold of difference, i.e., if the difference is less than the predetermined threshold, the compensation gain may be set to unity (i.e., no additional compensation).

[0107] Furthermore, in some embodiments according to the present disclosure, an expansion operation may be applied that may soften the volume boost.

[0108] The first expansion operation involves increasing the amount of decorrelation for size objects. Conventionally, when size objects are prebaked to an internal bed, the bed is traditionally decorrelated to preserve the timbre and naturalness of the voice. However, this may increase the possibility of loudness boost, since correlated signals may add acoustically within a cluster. Increasing the amount of decorrelation may reduce the loudness boost (albeit possibly at the expense of a change in timbre).

[0109] Thus, a method according to an embodiment of the present disclosure (e.g., method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100) may further include increasing the decorrelation between audio elements among the plurality of audio elements having a spatial size above a predetermined size threshold. The additional decorrelation may be applied specifically to the internal bed channel (i.e., to the audio element corresponding to the internal bed channel).

[0110] The second extension operation concerns subband gain estimation. Although the gains estimated / determined by the above methods (e.g., method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100) are wideband gains (i.e., the same gains are applied to all frequency bins), it may be useful to estimate gains from subbands (e.g., split based on ERB rate). The reason is that different subbands may play different perceptual roles, and subband-specific methods may provide higher frequency resolution to estimate loudness difference and object correlation.

[0111] Thus, in a method according to an embodiment of the present disclosure (eg, method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100), a compensation gain may be determined in each of a number of frequency subbands.

[0112] The third extension operation concerns loudness domain gain estimation. Although some of the above methods estimate the gain in the energy domain (related to loudness), the gain may be estimated / determined in the loudness domain to solve the loudness boost problem in a more straightforward way. It is well known to calculate loudness from the spectrum of an object. Thus, E o and E c The energy of volume L o and L c It is straightforward to calculate the respective volume gains by simply substituting

[0113] Thus, in a method according to an embodiment of the present disclosure (eg, method 300, 400 / 500, 600 / 700, 800 / 900, or 1000 / 1100), the energy metric may be a metric of loudness.

[0114] The present disclosure further relates to apparatuses including a processor and a memory coupled to the processor storing instructions for execution by the processor. The processor may be configured to perform any of the steps of the above-mentioned methods. It is understood that any description above of methods according to embodiments of the present disclosure applies to these apparatuses as well.

[0115] The present disclosure further relates to computer programs comprising instructions that, when executed by a processor, cause the computer programs to perform any of the steps of the above-mentioned methods, it being understood that any description above relating to the methods according to the embodiments of the present disclosure applies equally to these computer programs.

[0116] The present disclosure further relates to computer-readable storage media storing the aforementioned computer programs, it being understood that any description above relating to methods according to embodiments of the present disclosure applies equally to these computer-readable storage media.

[0117] As verified by simulation and listening, cluster-adaptive loudness normalization can significantly reduce the loudness boost, and adding target speaker layout-dependent loudness normalization can further improve the clustering quality.

[0118] Various aspects and implementations of the invention may be apparent from the following non-claimed enumerated example embodiments (EEE).

[0119] EEE1 is a method for processing audio content comprising a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; For a cluster in the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; determining, for at least one audio element in the cluster, a compensation gain based at least in part on the energy measures of the audio elements in the cluster; applying the compensation gain to the at least one audio element in the cluster; The present invention relates to a method comprising:

[0120] EEE2 further comprises determining whether the energy metric that an audio element contributes to cluster c is:

number

[0121] EEE3, for the cluster among the plurality of clusters, determining a spectrum of the cluster based on the respective spectra contributed by the audio elements to the cluster; determining, for each audio element in the cluster, as at least a portion of the compensation gain, an overall compensation gain for the cluster based at least in part on the energy measures of the audio elements in the cluster and the spectrum of the cluster; EEE1 or 2, comprising the steps of:

[0122] EEE4, for the cluster among the plurality of clusters, determining a first energy measure for the cluster as the sum of the energy measures contributed to the cluster by the audio elements in the cluster; determining a spectrum of the cluster based on the respective spectra contributed by the audio elements to the cluster; determining a second energy measure for the cluster based on the spectrum of the cluster; determining an overall compensation gain for the cluster based on the first energy measure and the second energy measure as at least a portion of the compensation gain for each audio element in the cluster; The method according to any one of claims 1 to 2, comprising:

[0123] EEE5: the first energy index of the cluster is given by:

number

number

number

[0124] EEE6 relates to the method as recited in EEE4 or EEE5, wherein the overall compensation gain of the cluster is determined as the square root of the ratio of the first energy measure and the second energy measure.

[0125] EEE7 provides, for a given audio element in a cluster among said plurality of clusters, determining a correlation metric between the given audio element and any of the plurality of audio elements; determining, as at least a portion of the compensation gain for the given audio element, an individual compensation gain for the given audio element based at least in part on the energy measures of the audio elements in the cluster and the correlation measures between the given audio element and any of the plurality of audio elements; EEE1 or 2, comprising the steps of:

[0126] EEE8 may, for a given audio element in a cluster among said plurality of clusters, determining a correlation metric between the given audio element and any of the plurality of audio elements; determining a third energy measure for the given audio element as a weighted sum of the energy measures that the audio elements contribute to the cluster, the weighted sum of the energy measures being based on respective correlation measures between the respective audio elements and the given audio element; determining a fourth energy measure for the given audio element as a weighted sum of geometric means of the energy measure that the given audio element contributes to the cluster and each energy measure that the audio elements in the plurality of audio elements excluding the given audio element contribute to the cluster, over any audio elements in the plurality of audio elements excluding the given audio element, wherein weights of the geometric means are based on respective correlation measures between the each audio element and the given audio element; determining an individual compensation gain for the given audio element based on the third energy measure and the fourth energy measure as at least a part of the compensation gain for the given audio element; EEE1 or 2, comprising the steps of:

[0127] EEE9 states that the correlation measure between the given audio element and any of the plurality of audio elements is given by: [Number] where indexes o and u indicate the given audio element and one of the plurality of audio elements, and X o is the spectrum of the given audio element, and X u is the spectrum of one of the plurality of audio elements, E o is the energy of the given audio element, Eu is the energy of one of the plurality of audio elements, and the third energy measure is given by:

number

number

[0128] For EEE10, the individual compensation gain is given by:

number

[0129] EEE11, for the cluster among the plurality of clusters, determining a respective individual compensation gain for each audio element in the cluster; applying each individual compensation gain to the audio elements in the cluster to obtain an individual compensation audio element; determining a spectrum of the cluster based on the respective spectra contributed by the individual compensation audio elements to the cluster; determining, for each individual compensation audio element in the cluster, as at least a portion of the compensation gain, an overall compensation gain for the cluster based at least in part on the energy measures of the individual compensation audio elements in the cluster and the spectrum of the cluster; The present invention relates to a method according to any one of claims EEE7 to EEE10, comprising:

[0130] EEE12, for the cluster among the plurality of clusters, determining a respective individual compensation gain for each of the audio elements in the cluster; applying each individual compensation gain to the audio elements in the cluster to obtain an individual compensation audio element; determining a fifth energy measure for the cluster as the sum of the energy measures that the individual compensation audio elements in the cluster contribute to the cluster; determining a spectrum of the cluster based on the respective spectra contributed by the individual compensation audio elements to the cluster; determining a sixth energy index of the cluster based on the spectrum of the cluster; determining an overall compensation gain for the cluster based on the fifth energy measure and the sixth energy measure as at least a portion of the compensation gain for each individual compensation audio element in the cluster; The present invention relates to a method according to any one of claims EEE7 to EEE10, comprising:

[0131] EEE13, for a speaker on which at least one of the clusters is rendered, determining an energy metric that each of said audio elements contributes to the output of said speaker; determining a spectrum of the output of the speaker based on the respective spectra that the audio elements contribute to the output of the speaker; determining an overall compensation gain for the speaker based at least in part on the energy measure that the audio element contributes to the output of the speaker and the spectrum of the output of the speaker; The present invention relates to a method according to any one of claims 1 to 12, further comprising:

[0132] The EEE14 performs the following steps for a speaker on which at least one of the clusters is rendered: determining an energy metric that each of said audio elements contributes to the output of said speaker; determining a seventh energy metric of the output of the speaker based on the respective energy metric that the audio elements contribute to the output of the speaker; determining a spectrum of the output of the speaker based on the respective spectra that the audio elements contribute to the output of the speaker; determining an eighth energy measure of the output of the speaker based on the spectrum of the output of the speaker; determining an overall compensation gain for the speaker based on the seventh energy metric and the eighth energy metric; The method according to any one of EEE1 to EEE12, further comprising:

[0133] EEE15, the seventh energy index is given by:

number

number

number

[0134] EEE16 relates to the method according to EEE14 or 15, wherein the overall compensation gain of the loudspeaker is determined as the square root of the ratio of the seventh energy index and the eighth energy index.

[0135] EEE17 relates to the method according to any one of EEE1 to EEE16, wherein the compensation gain is determined for each frame or for each group of frames of the audio content.

[0136] EEE18 further comprises the step of: clustering the plurality of audio elements into the plurality of clusters, clustering the plurality of audio elements into a plurality of intermediate clusters; clustering the intermediate clusters into the clusters; The present invention relates to a method according to any one of claims 1 to 17, comprising:

[0137] EEE19 further comprises the steps of: applying a dynamic range compressor or limiter to said determined compensation gains before applying said compensation gains to the respective audio elements; The method according to any one of claims 1 to 18, further comprising:

[0138] setting said compensation gain to unity depending on whether the difference between the expected energy and the actual energy of each cluster is less than a predetermined threshold of said difference; The method of any one of EEE1-19 further comprises:

[0139] EEE21 includes a step of increasing decorrelation between audio elements among said plurality of audio elements having a spatial size exceeding a predetermined threshold value for size; The method of any one of EEE1-20 further comprises:

[0140] EEE22 relates to the method according to any one of EEE1 to EEE21, wherein the compensation gains are determined in each of a plurality of frequency subbands.

[0141] EEE23 relates to the method according to any one of EEE1 to 22, wherein said energy measure is a measure of loudness.

[0142] EEE24 relates to an apparatus including a processor and a memory coupled to the processor storing instructions for execution by the processor, the processor being configured to perform method steps of the methods described in any one of EEE1 to EEE23.

[0143] EEE25 relates to a computer program comprising instructions which, when executed by a processor, cause the processor to perform a method for processing audio content as described in any one of EEE1 to EEE23.

[0144] EEE26 relates to a computer readable medium storing the computer program described in EEE25.

Claims

1. 1. A method for processing audio content comprising a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; For a cluster in the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; determining a compensation gain for at least one audio element in the cluster based at least in part on the energy measures of the audio elements in the cluster; applying the compensation gain to the at least one audio element in the cluster; Including, The energy measure that an audio element contributes to the cluster c is [0046] where E o is the energy of the audio element, and g oc is the element-cluster gain of the audio element o, where the element-cluster gain is the gain with which the audio element o is rendered into the cluster c; The step of determining a compensation gain comprises: For the cluster in the plurality of clusters, determining a spectrum of the cluster based on the respective spectra contributed by the audio elements to the cluster; determining, for each audio element in the cluster, as at least a portion of the compensation gain, an overall compensation gain for the cluster based at least in part on the energy measures of the audio elements in the cluster and the spectrum of the cluster; A method comprising:

2. A method for processing audio content including a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; For a cluster in the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; determining a compensation gain for at least one audio element in the cluster based at least in part on the energy measures of the audio elements in the cluster; applying the compensation gain to the at least one audio element in the cluster; Including, The energy measure that an audio element contributes to the cluster c is [0046] where E o is the energy of the audio element, g oc is the element-cluster gain of the audio element o, the element-cluster gain being the gain with which the audio element o is rendered into the cluster c, The step of determining a compensation gain comprises: For the cluster in the plurality of clusters, determining a first energy measure for the cluster as the sum of the energy measures contributed to the cluster by the audio elements in the cluster; determining a spectrum of the cluster based on the respective spectra contributed by the audio elements to the cluster; determining a second energy measure for the cluster based on the spectrum of the cluster; determining an overall compensation gain for the cluster based on the first energy measure and the second energy measure as at least a portion of the compensation gain for each audio element in the cluster; A method comprising:

3. The first energy index of the cluster is given by: [0047] and / or the second energy indicator is given by: [0048] where index o denotes each audio element in the cluster, [0049] is the spectrum of the cluster, and X o is the spectrum of each of the audio elements, * The method of claim 2, wherein is the complex conjugate of ■.

4. The method of claim 2 or 3, wherein the overall compensation gain of the cluster is determined as the square root of the ratio of the first energy measure and the second energy measure.

5. A method for processing audio content comprising a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; For a cluster in the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; determining a compensation gain for at least one audio element in the cluster based at least in part on the energy measures of the audio elements in the cluster; applying the compensation gain to the at least one audio element in the cluster; Including, The energy measure that an audio element contributes to the cluster c is [0046] where E o is the energy of the audio element, g oc is the element-cluster gain of the audio element o, the element-cluster gain being the gain with which the audio element o is rendered into the cluster c, The step of determining a compensation gain comprises: For a given audio element in a cluster in the plurality of clusters, determining a correlation metric between the given audio element and any of the plurality of audio elements; determining, as at least a portion of the compensation gain for the given audio element, an individual compensation gain for the given audio element based at least in part on the energy measures of the audio elements in the cluster and the correlation measures between the given audio element and any of the plurality of audio elements; A method comprising:

6. A method for processing audio content comprising a plurality of audio elements, the method comprising: clustering the plurality of audio elements into a plurality of clusters of audio elements; For a cluster in the plurality of clusters, determining, for each audio element in the cluster, an energy metric that the audio element contributes to the cluster; determining a compensation gain for at least one audio element in the cluster based at least in part on the energy measures of the audio elements in the cluster; applying the compensation gain to the at least one audio element in the cluster; Including, The energy measure that an audio element contributes to the cluster c is [0046] where E o is the energy of the audio element, g oc is the element-cluster gain of the audio element o, the element-cluster gain being the gain with which the audio element o is rendered into the cluster c, The step of determining a compensation gain comprises: For a given audio element in a cluster in the plurality of clusters, determining a correlation metric between the given audio element and any of the plurality of audio elements; determining a third energy measure for the given audio element as a weighted sum of the energy measures that the audio element contributes to the cluster, the weighting of the energy measure being based on respective correlation measures between each audio element in the plurality of audio elements and the given audio element; determining a fourth energy measure for the given audio element as a weighted sum of geometric means of the energy measure that the given audio element contributes to the cluster and each energy measure that the audio element in the plurality of audio elements excluding the given audio element contributes to the cluster, across any audio elements in the plurality of audio elements excluding the given audio element, wherein weights of the geometric means are based on respective correlation measures between each audio element in the plurality of audio elements and the given audio element; determining an individual compensation gain for the given audio element based on the third energy measure and the fourth energy measure as at least a part of the compensation gain for the given audio element; A method comprising:

7. 7. The method according to claim 5 or 6, wherein the individual compensation gain of the given audio element is determined such that the greater the correlation index between the given audio element and any of the plurality of audio elements, the smaller the individual compensation gain of the given audio element.

8. The correlation measure between the given audio element and any of the plurality of audio elements is given by: [Number 50] where indexes o and u indicate the given audio element and one of the plurality of audio elements, and X o is the spectrum of the given audio element, and X u is the spectrum of one of the plurality of audio elements, E o is the energy of the given audio element, E u is the energy of one of the plurality of audio elements, and the third energy measure is given by: [0051] and / or the fourth energy indicator is given by: [0052] The method according to claim 6.

9. The individual compensation gain is given by: [0,53] The method according to claim 8.

10. For the cluster in the plurality of clusters, determining a respective individual compensation gain for each audio element in the cluster; applying each individual compensation gain to the audio elements in the cluster to obtain an individual compensation audio element; determining a spectrum of the cluster based on the respective spectra contributed by the individual compensation audio elements to the cluster; determining, for each individual compensation audio element in the cluster, as at least a portion of the compensation gain, an overall compensation gain for the cluster based at least in part on the energy measures of the individual compensation audio elements in the cluster and on the spectrum of the cluster; The method according to any one of claims 5 to 9, comprising:

11. For the cluster in the plurality of clusters, determining a respective individual compensation gain for each of the audio elements in the cluster; applying each individual compensation gain to the audio elements in the cluster to obtain an individual compensation audio element; determining a fifth energy measure for the cluster as the sum of the energy measures that the individual compensation audio elements in the cluster contribute to the cluster; determining a spectrum of the cluster based on the respective spectra contributed by the individual compensation audio elements to the cluster; determining a sixth energy index of the cluster based on the spectrum of the cluster; determining an overall compensation gain for the cluster based on the fifth energy measure and the sixth energy measure as at least a portion of the compensation gain for each individual compensation audio element in the cluster; The method according to any one of claims 5 to 9, comprising:

12. For a speaker on which at least one of the clusters is rendered, determining an energy metric that each of said audio elements contributes to the output of said speaker; determining a spectrum of the output of the speaker based on the respective spectra that the audio elements contribute to the output of the speaker; determining an overall compensation gain for the speaker based at least in part on the energy measure that the audio element contributes to the output of the speaker and the spectrum of the output of the speaker; The method of any one of claims 1 to 11, further comprising:

13. For a speaker on which at least one of the clusters is rendered, determining an energy metric that each of said audio elements contributes to the output of said speaker; determining a seventh energy metric of the output of the speaker based on the respective energy metric that the audio elements contribute to the output of the speaker; determining a spectrum of the output of the speaker based on the respective spectra that the audio elements contribute to the output of the speaker; determining an eighth energy measure of the output of the loudspeaker based on the spectrum of the output of the loudspeaker; determining an overall compensation gain for the speaker based on the seventh energy metric and the eighth energy metric; The method of any one of claims 1 to 11, further comprising:

14. The seventh energy index is given by: [0,54] Element - Speaker Gain g os for an audio element o among the plurality of audio elements and the speaker s, and the spectrum of the output of the speaker is given by: [0.55] The index c indicates the cluster, and X o denotes the spectrum of a given audio element o, and g cs is the cluster-speaker gain for cluster c and speaker s, and g oc is the element-cluster gain for cluster c and audio element o within the cluster, and / or the eighth energy measure is given by: [0,56] The method of claim 13.

15. 15. The method of claim 13 or 14, wherein the overall compensation gain of the loudspeaker is determined as the square root of the ratio of the seventh energy index and the eighth energy index.

16. The method according to any one of the preceding claims, wherein the compensation gain is determined for each frame or for each group of frames of the audio content.

17. The step of clustering the plurality of audio elements into the plurality of clusters comprises: clustering the plurality of audio elements into a plurality of intermediate clusters; clustering the intermediate clusters into the clusters; The method according to any one of claims 1 to 16, comprising:

18. applying a dynamic range compressor or limiter to said determined compensation gains before applying said compensation gains to the respective audio elements; The method of any one of claims 1 to 17, further comprising:

19. setting said compensation gain to unity depending on whether the difference between expected energy and the actual energy of each cluster is less than a predetermined threshold of said difference; The method of any one of claims 1 to 18, further comprising:

20. increasing decorrelation between audio elements among said plurality of audio elements having a spatial size exceeding a predetermined size threshold; The method of any one of claims 1 to 19, further comprising:

21. The method of any one of claims 1 to 20, wherein the compensation gain is determined in each of a plurality of frequency subbands.

22. The method of any one of claims 1 to 21, wherein the energy measure is a measure of loudness.

23. An apparatus comprising a processor and a memory coupled to said processor storing instructions for execution by said processor, said processor configured to perform the method steps of the method of any one of claims 1 to 22.

24. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method for processing audio content according to any one of claims 1 to 22.

25. 25. A computer readable medium storing a computer program according to claim 24.

Citation Information

Patent Citations

  • Audio signal processing method and apparatus

    JP2015531078A

  • Audio object clustering based on renderer-aware perceptual difference

    WO2018017394A1