Metadata-preserving audio object clustering

Abstract

To provide example embodiments disclosed herein which relate to audio object clustering.SOLUTION: A method for metadata-preserved audio object clustering is disclosed. The method comprises classifying a plurality of audio objects into a number of categories on the basis of information to be preserved in metadata associated with the plurality of audio objects. The method further comprises assigning a predetermined number of clusters to the categories and allocating an audio object in each of the categories to at least one of the clusters according to the assigning. A corresponding system and a computer program product are also disclosed.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Chinese Patent Application No. 201410765578.6, filed December 11, 2014, and U.S. Provisional Patent Application No. 62 / 100,183, filed January 6, 2015, the contents of each of which are incorporated herein by reference in their entirety.

[0002] technology FIELD OF THE DISCLOSURE Exemplary embodiments disclosed herein relate generally to audio content processing, and more particularly to methods and systems for audio object clustering that allow metadata to be preserved. [Background technology]

[0003] The advent of object-based audio has significantly increased the amount of audio data and the complexity of rendering this data in high-end playback systems. For example, a movie soundtrack may contain many different sound elements, dialogue, noises and sound effects that correspond to the images on the screen. These soundtracks also emanate from different positions on the screen and combine with background music and ambient effects to create a holistic auditory experience. Accurate reproduction requires that sounds are reproduced in a way that corresponds as closely as possible to what is shown on the screen in terms of source position, intensity, movement and depth. Object-based audio represents a significant improvement over traditional channel-based audio systems, which send audio content in the form of speaker feeds to individual speakers within the listening environment and are therefore relatively limited in terms of the spatial reproduction of specific audio objects.

[0004] The introduction of digital cinema and the development of three-dimensional ("3D") content have created new standards for sound, such as the incorporation of multi-channel audio allowing greater creativity for content creators and a more immersive and realistic auditory experience for audiences. It is essential to expand beyond traditional speaker feeds and channel-based audio as a means of delivering spatial audio. Furthermore, there has been much interest in model-based audio descriptions that allow listeners to select a desired playback configuration by using audio that is rendered specifically for the chosen configuration. Spatial presentation of sound makes use of audio objects. An audio object is an audio signal with associated parametric source descriptions, such as apparent source position (e.g., 3D coordinates), apparent source width, and other parameters. As a further advancement, next-generation spatial audio (also called "adaptive audio") formats are being developed by including a mix of audio objects and traditional channel-based speaker feeds (audio beds) along with position metadata for the audio objects.

[0005] As used herein, the term "audio object" refers to a discrete audio element that exists in the sound field for a defined duration, while the term "audio bed" or "bed" refers to an audio channel that is intended to be reproduced at a predefined, fixed speaker position.

[0006] In some soundtracks, there may be several (e.g., 7, 9 or 11) bed channels containing audio. Furthermore, based on the capabilities of the authoring system, there may be dozens or even hundreds of individual audio objects that are combined during rendering to create a spatially diverse and immersive audio experience. In other distribution and transmission systems, there may be a large enough available bandwidth to transmit all the audio beds and objects with little or no audio compression. However, in some cases, such as Blu-ray Disc, broadcast (cable, satellite, terrestrial), mobile (3G and 4G) and over-the-top (OTT or Internet) distribution, there may be significant limitations in the available bandwidth to digitally transmit all of the bed and object information generated at the time of authoring. Although audio coding methods (lossless or lossy) may be applied to the audio to reduce the required bandwidth, the audio coding may not be sufficient to reduce the bandwidth required to transmit the audio over very limited networks, such as mobile 3G and 4G networks.

[0007] Several conventional methods have been developed to reduce the number of input objects to a smaller set of output objects by clustering. Generally, some clustering process requires that metadata such as size, zone mask and snapping should be pre-rendered to the internal channel layout. The clustering of audio objects is based only on the spatial location of the audio objects, and the output objects contain only positional metadata. This kind of output objects may not work well for some playback systems, since the loss of metadata may break the expected artistic intent.

[0008] It should not be assumed that the subject matter discussed in the Background section is prior art merely by virtue of its disclosure in the Background section. Similarly, it should not be assumed that any problem mentioned in or related to the subject matter of the Background section has been previously recognized in the prior art. The subject matter in the Background section is merely representative of various approaches, which themselves may be example embodiments. Summary of the Invention [Problem to be solved by the invention]

[0009] To address the above and other potential problems, the illustrative embodiments propose a method and system for audio object clustering in which metadata is preserved. [Means for solving the problem]

[0010] In one aspect, the exemplary embodiments provide a method for audio object clustering in which metadata is preserved. The method includes classifying a plurality of audio objects into categories based on information to be preserved in metadata associated with the plurality of audio objects. The method further includes assigning a number of clusters to the categories and allocating audio objects in each of the categories to at least one of the clusters based on the assignment. Embodiments in this regard further include corresponding computer program products.

[0011] In another aspect, an exemplary embodiment provides a system for audio object clustering in which metadata is preserved, the system including an audio object classification unit configured to classify a plurality of audio objects into categories based on information to be preserved in metadata associated with the plurality of audio objects, a cluster assignment unit configured to assign a predetermined number of clusters to the categories, and an audio object allocation unit configured to allocate audio objects in each of the categories to at least one of the clusters based on the assignment.

[0012] Through the above description, it will be understood that according to the exemplary embodiment disclosed herein, the input audio objects are classified into corresponding categories depending on the information to be stored in the metadata, so that different metadata to be stored or unique combinations of metadata to be stored are associated with different categories. After clustering, for an audio object in a category, it is less likely that it will be mixed with an audio object associated with different metadata. In this regard, the metadata of the audio objects can be preserved after clustering. Other advantages achieved by the exemplary embodiment will be apparent from the following description. [Brief description of the drawings]

[0013] The above and other objects, features and advantages of the embodiments will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which several exemplary embodiments are shown by way of illustration and not limitation. [Figure 1] 1 is a flowchart of a method for audio object clustering with preserved metadata according to an example embodiment. [Diagram 2]FIG. 1 is a schematic diagram for an audio object clustering process according to an example embodiment. [Diagram 3] 1 is a block diagram of a system for audio object clustering with metadata preservation according to an example embodiment. [Figure 4] 1 is a block diagram of an exemplary computer system suitable for implementing the embodiments.Throughout the drawings, like or corresponding reference characters refer to like or corresponding parts. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The principles of the exemplary embodiments will now be described with reference to various exemplary embodiments illustrated in the drawings, it being understood that the illustrations of these embodiments are merely to enable those skilled in the art to better understand and further implement the exemplary embodiments, and are not intended to limit the scope in any manner.

[0015] As mentioned above, due to limitations in encoding / decoding rates and transmission bandwidth, the number of audio objects used to generate adaptive audio content may be reduced by clustering. Besides the metadata describing its spatial location, an audio object usually has other metadata describing its attributes, such as size, zone mask, snapping, and content type. Each of these attributes describes the artistic intent of how the audio object should be treated when rendered. However, in some conventional methods, after the audio objects are clustered, only the position metadata remains. Other metadata may be pre-rendered into the internal channel layout, such as in a 7.1.2 or 7.1.4 system, but that does not work well for all systems. The artistic intent of the audio object may be broken when rendered, especially when the audio object is downmixed to, for example, a 5.1 or 7.1 system.

[0016] Take the metadata "Zone Mask" for example. It has several modes, each mode defines an area where an audio object should not be rendered. One mode of the Zone Mask is "No Sides" which describes that the side speakers should be masked when rendering that audio object. If an audio object at spatial location z=1 is rendered to a 5.1 system with the metadata "No Sides" by using traditional clustering methods, the side speakers may be activated in the 5.1 rendering because the sound from the ceiling speakers may be convoluted to the sides. To address this issue, the metadata "Zone Mask" should be preserved in the clustering process so that it can be correctly handled by the audio renderer.

[0017] In another example, dialogue objects may be expected to be separated from other objects after clustering. This may have many benefits for subsequent audio object processing. For example, in subsequent audio processing such as dialogue enhancement, the separated dialogue object clusters can be easily enhanced by simply applying gain(s). Otherwise, it may be very difficult to separate the dialogue objects when they are mixed with other objects in a cluster. In dialogue replacement applications, the dialogues in each language may be completely separated from each other. For such purpose, in the clustering process, the dialogue objects should be preserved and allocated to separate individual clusters.

[0018] Additionally, audio objects may be associated with metadata describing their rendering mode, for example to be rendered as left-sum / right-sum (Lt / Rt) or as binaural with head-related transfer functions (HRTFs) when processed in a headphone renderer. These rendering modes are also expected to be preserved after clustering in order to produce the best rendering results.

[0019] Therefore, it is desirable to allow metadata to be preserved in audio object clustering to achieve a better audio experience.The exemplary embodiments disclosed herein propose a method and system for object clustering in which metadata is preserved.

[0020] Reference is first made to Figure 1, which illustrates a flow chart of a method 100 for metadata-preserved audio object clustering according to an exemplary embodiment.

[0021] In S101, a plurality of audio objects are classified into categories based on information to be stored in metadata associated with the plurality of audio objects. These audio objects are provided as input, and there may be tens, hundreds, or even thousands of input audio objects.

[0022] As used herein, information to be stored in the metadata associated with each audio object may indicate the processing intent when the audio object is rendered. The information may describe how the audio object should be processed when it is rendered. In some embodiments, the information may include one or more of the size information, zone mask information, snap information, content type, or rendering mode of the audio object. The size information may be used to indicate the spatial area or volume occupied by the audio object. The zone mask information indicates the mode of a zone mask that defines the areas where the audio object should not be rendered. For example, the zone mask information may indicate modes such as "no sides", "surround only", "front only", etc. The snap information indicates whether the audio object should be panned directly to the nearest speaker.

[0023] It should be noted that several examples of information to be stored in the metadata are described, and depending on the preferences of the user or audio engineer, other information contained in the metadata may also be considered in the audio object classification (non-limiting examples include spatial location, spatial width, etc.) In some embodiments, all information in the metadata associated with an audio object may be considered.

[0024] The number of categories may depend on the information in the metadata of those audio objects and may be one or more. In an embodiment, audio objects without information to be stored may be classified into one category, and audio objects with different information to be stored may be classified into different categories. That is, depending on the different information to be stored, corresponding audio objects are classified into different categories. Alternatively, a category may represent a unique combination of different information to be stored in the metadata. All other audio objects without information of interest may be included in one category or multiple categories in some cases. The scope of the example embodiments is not limited in this respect.

[0025] The categories may be provided by manual assignment, automatic assignment or a combination thereof. For example, a user or an audio engineer may label audio objects associated with different types of metadata with different flags, and then the labeled audio objects may be classified into different categories according to the flags. As another example, the information to be stored in the metadata may be identified automatically. A user or an audio engineer may pre-configure their preferences or expectations, such as separating dialogue objects, separating different dialogue languages, and / or separating different modes of zone masks. Depending on the pre-configuration, the audio objects may be classified into different categories.

[0026] Suppose there are O audio objects. In the classification process, the information to be stored in the metadata of the audio objects may be derived from (1) manual labels of the metadata given by user input, e.g., zone masks or snaps or content type or language labels, and / or (2) automatic identification / labeling of the metadata, e.g., but not limited to, content type identification information. The number N of possible categories may be determined according to the derived information, each category consisting of a unique combination of information to be stored. After classification, each audio object has an associated category identification information n o It may have.

[0027] Referring to Figure 2, a schematic diagram of audio object clustering is shown. As shown in Figure 2, multiple input audio objects are classified into five categories, categories 0 to 4, based on the information to be preserved in the metadata. An example of the categories may be given as follows: Category 0: all audio objects with no information to be saved; Category 1: Music object, no zone mask; Category 2: Sound effect objects, with zone mask "Surround only"; Category 3: English dialogue objects; Category 4: Spanish dialogue objects, with "Ahead Only" zone mask.

[0028] The input audio objects may contain one or more frames. A frame is the processing unit for audio content, and the duration of a frame may vary and may depend on the configuration of the audio processing system. Since the audio objects to be classified may change in time for different frames and their metadata may also change, the value of the number of categories may also change over time. Categories representing different kinds of information to be preserved may be predefined by the user or by default, in which case the input audio objects in one or more frames may be classified into the predefined categories based on that information. In the subsequent processing, the categories with classified audio objects may be taken into account and those without audio objects may be ignored. For example, when there are no audio objects without information to be preserved in Fig. 2, the corresponding category 0 may be omitted. It is considered that the number of audio objects classified into each category may change over time.

[0029] In S102, a predetermined number of clusters is assigned to the categories. The predetermined number may be greater than 1 and may depend on the transmission bandwidth and the encoding / decoding rate of the audio processing system. There may be a trade-off between the transmission bandwidth (and / or the encoding rate and / or the decoding rate) and the error criterion of the output audio objects. For example, the predetermined number may be 11 or 16. Other values, such as 5, 7 or 20, may also be determined, and the scope of the exemplary embodiments is not limited in this respect.

[0030] In some embodiments, the predetermined number may not vary within the same processing system, while in other embodiments, the predetermined number may vary for different audio files to be processed.

[0031] In the exemplary embodiments disclosed herein, audio objects are first classified into categories in S101 according to metadata, such that each category may represent a different information or a unique combination of different information to be stored. The audio objects in those categories may then be clustered in a subsequent process. There may be various approaches to assign / allocate a predefined overall number of clusters to the categories. In some exemplary embodiments, the overall number of clusters is predetermined and fixed, so that it is possible to determine the number of clusters to be assigned to each category before clustering the audio objects. Some exemplary embodiments will now be discussed.

[0032] In an exemplary embodiment, cluster assignment may depend on the importance of the plurality of audio objects. Specifically, the predetermined number of audio objects from the plurality of audio objects may be first determined based on the importance of each audio object relative to other audio objects, and then a distribution of the predetermined number of audio objects among the categories may be determined. The predetermined number of clusters are assigned to the categories correspondingly according to the distribution.

[0033] The importance of each audio object may be related to one or more of the type of content, the fractional loudness level, or the energy level of that audio object. An audio object of high importance may represent that the audio object is perceptually prominent among the input audio objects, for example due to its fractional loudness level or energy level. In some use cases, one or more types of content may be considered important, and the corresponding audio object may then be given high importance. For example, dialogue objects may be assigned higher importance. It should be noted that there are many other ways to determine or define the importance of each audio object. For example, the importance level of some audio objects may be specified by a user. The scope of the example embodiments is not limited in this respect.

[0034] Suppose there is a predetermined overall number of clusters, M. In a first step, up to M most important audio objects among the input audio objects are selected. Since all input audio objects are classified into corresponding categories in S101, in a second step, a distribution of those M important audio objects among the categories may be determined. Based on how many of those M audio objects are distributed into a category, an equal number of clusters may be assigned to that category.

[0035] Referring to Fig. 2, for example, eleven of the most important audio objects (shown as circles 201) are determined from a plurality of input audio objects (shown as a collection of circles 201 and 202). After classifying all the input audio objects into five categories, categories 0 to 4, it can be seen from Fig. 2 that four most important audio objects are classified into category 0, three most important audio objects are classified into category 1, one most important audio object is classified into category 2, two most important audio objects are classified into category 3, and one most important audio object is classified into category 4. As a result, as shown in Fig. 2, categories 0 to 4 are assigned 4, 3, 1, 2, and 1 clusters, respectively.

[0036] It should be noted that the above example of importance criteria according to this exemplary embodiment of the exemplary embodiments may not be very strict, i.e. it is not essential that the most important audio objects are selected. In some embodiments, an importance threshold may be configured. The predetermined number of audio objects may be randomly selected from among the audio objects whose importance is higher than a threshold.

[0037] Besides importance criteria, cluster assignment may also be performed based on reducing the overall spatial distortion for the categories, i.e., the predetermined number of clusters may be assigned to the categories based on reducing or even minimizing the overall spatial distortion for the categories.

[0038] In an exemplary embodiment, the overall spatial distortion for the categories may include a weighted sum of the individual spatial distortions of those categories. The weight of the corresponding category may represent the importance of that category or the importance of the information associated with that category to be preserved. For example, a category with higher importance may have a higher weight. In another embodiment, the overall spatial distortion for the categories may include the maximum spatial distortion among the individual spatial distortions of those categories. It should be considered that it is not necessary that only the maximum is selected, and in some embodiments, other spatial distortions among the categories, such as the second largest spatial distortion, the third largest spatial distortion, etc. may be considered as the overall spatial distortion.

[0039] The spatial distortion for each category may be represented by the distortion level of the audio objects contained in that category, and the distortion level of each audio object may be measured by the difference between its original spatial location and its clustered location. In general, the clustered location of an audio object depends on the spatial location of the cluster(s) to which it is assigned. In this sense, the spatial distortion for each category is related to the original spatial location of each audio object in that category and the spatial location of the cluster(s). The original spatial locations of the audio objects may be included in the metadata of the audio objects and may consist, for example, of three Cartesian coordinates (or may equally consist, for example, of polar coordinates or cylindrical and spherical coordinates, homogeneous coordinates, line number coordinates, etc.). In an embodiment, to calculate the spatial distortion for each category, the reconstructed spatial location of each audio object in the category may be determined based on the spatial location of the cluster(s). The spatial distortion for each category may then be calculated based on the distance between the original spatial location of each audio object in the category and the reconstructed spatial location of the audio object. The reconstructed spatial location of the audio object is the spatial location of the audio object represented by one or more corresponding spatial clusters. One exemplary approach for determining the reconstructed spatial location is described below.

[0040] To obtain an overall spatial distortion, the spatial distortion for different numbers of clusters may first be calculated for each category. There are many approaches to determine the spatial distortion for a category of audio objects. One approach is given below as an example. It should be noted that other existing ways of measuring the spatial distortion of an audio object (and thus of a category) may also be applied.

[0041] For category n, spatial location

number

number

number

number

[0042] Once the spatial distortion for each category is obtained, in one embodiment, the overall spatial distortion for the categories may be determined as a weighted sum of the individual spatial distortions of the categories as described above. For example, the overall spatial distortion may be determined as follows:

number

[0043] In another embodiment, the overall spatial distortion for the categories may be determined as the maximum spatial distortion among the individual spatial distortions of those categories. For example, the overall spatial distortion may be determined as follows:

number

number

[0044] The input audio objects are generally within one frame of the audio signal. Due to the typical dynamic nature of the audio signal, and considering that the number of audio objects varies in each category, the number of clusters assigned to each category may typically vary over time. Since the changed number of clusters for each category may cause some instability issues, a modified spatial distortion that takes into account the cluster number consistency is utilized in the cost metric. As a result, the cost metric may be defined as a function of time. Specifically, the spatial distortion for each category is further based on the difference between the number of clusters assigned to that category in the current frame and the number of clusters assigned to that category in the previous frame. In this regard, the overall spatial distortion in equation (2) may be modified as follows:

number

number

[0045] If the number of clusters assigned to a category changes in the current frame compared to the previous spatial distortion, the modified spatial distortion may be increased to prevent the change in the number of clusters. In one embodiment, f(D n (M n ),M n ,M n ') may be determined as follows:

number

[0046] Since reducing the number of clusters of a category is more likely to introduce spatial instability than increasing the number of clusters, in another embodiment, f(D n (M n ),M n ,M n ') may be determined as follows:

number

[0047] In the above description, for cluster assignment based on reducing the overall spatial distortion, determining the optimal number of clusters for each category may involve a large amount of computational effort. In order to efficiently determine the number of clusters for each category, in one embodiment, an iterative process is proposed. That is, the optimal number of clusters for each category is estimated by maximizing the cost reduction at each iteration step of the cluster assignment process. Thereby, the overall spatial distortion for the category may be iteratively reduced or even minimized.

[0048] By iterating successively from 1 to a predetermined number of clusters M, at each iteration step, one or more clusters are assigned to the category that needs it most. Denote the overall spatial distortion at the (m-1)th and mth iteration steps as Cost(m-1) and Cost(m). At the mth iteration step, one or more new clusters may be assigned to the category n* that can reduce the overall spatial distortion the most. Thus, n* may be determined by expanding or maximizing the reduction in the overall spatial distortion. This may be expressed as follows:

number

[0049] For an overall spatial distortion obtained by a weighted sum of all spatial distortions of the categories, the successive iteration process may be based on the difference between the spatial distortion for a category in the current iteration and the previous iteration. In each iteration, at least one cluster may be assigned to a category such that, when the category is assigned together with the at least one cluster, its spatial distortion in the current iteration is sufficiently lower (according to a first predetermined level) than its spatial distortion in the previous iteration. In one embodiment, the at least one cluster may be assigned to the category that has the most reduced spatial distortion when assigned together with the at least one cluster. For example, in this embodiment, n* may be determined as follows:

number

[0050] With the overall spatial distortion determined as the maximum spatial distortion among all categories, the successive iterations may be based on the amount of spatial distortion for a category in a previous iteration. In each iteration, at least one cluster may be assigned to a category that has a spatial distortion higher than a second predetermined level in the previous iteration. In one embodiment, the at least one cluster may be assigned to the category with the maximum spatial distortion in the previous iteration. For example, in this embodiment, n* may be determined as follows:

number

[0051] Note that the decisions given in equations (9) and (10) may be used jointly in one iterative process. For example, in one iteration, equation (9) may be used to assign new cluster(s) in that iteration. In another iteration, equation (10) may be used to assign other new cluster(s).

[0052] Two ways of cluster assignment are described above: one based on the importance of audio objects, the other based on reducing the overall spatial distortion. Additionally or alternatively, user input may also be used to guide the cluster assignment. This may greatly improve the flexibility of the clustering process, since users may have different requirements for different content for different use cases. In some embodiments, the cluster assignment may be further based on one or more of: a first threshold for the number of clusters to be assigned to each category, a second threshold for the spatial distortion for each category, or the importance of each category relative to other categories.

[0053] The first threshold may be predefined for the number of clusters to be assigned to each category. The first threshold may be a predefined minimum or maximum number of clusters for each category. For example, the user may specify that a category should have a certain minimum number of clusters. In this case, at least the specified number of clusters should be assigned to the category during the allocation process. If a maximum threshold is set, at most the specified number of clusters can be assigned to the category. The second threshold may be set to ensure that the spatial distortion for a category is reduced to a reasonable level. The importance of each category may also be specified by the user or may be determined based on the importance of the audio objects classified in that category.

[0054] In some cases, the spatial distortion for a category may be high after cluster assignments are made, which may introduce audible artifacts. To address this issue, in some embodiments, at least one audio object in a category may be reclassified to another category based on the spatial distortion for that category. In one example embodiment, if the spatial distortion for one of the categories is higher than a predefined threshold, some audio objects in that category may be reclassified to another category until the spatial distortion is less than (or equal to) the threshold. In some examples, audio objects may be reclassified to a category that includes audio objects with no information to be preserved in metadata, such as category 0 in FIG. 2. In some embodiments, where the cluster assignment is based on minimizing the overall spatial distortion in a successive iteration process, the object reallocation also involves minimizing the maximum spatial distortion dis(o) at each iteration until the spatial distortion criterion is met for that category. n (I C n (1), C n (2),…,C n (M n )}) may be an iterative process in which audio objects with

[0055] Due to the typical dynamic nature of audio signals, the importance or spatial location (and hence spatial distortion) of audio objects change over time. As a result, cluster assignments may be time-varying, in which case the number of clusters assigned to each category may change over time. In this case, the category identity associated with cluster m may change over time. In particular, cluster m may represent one language (e.g. Spanish) during a first frame, while the category identity and hence the language may change (e.g. English) for a second frame. This is in contrast to legacy channel-based systems, where the language is statically tied to the channel rather than changing dynamically.

[0056] Cluster assignment in S102 was described above.

[0057] Returning to reference to FIG. 1, at S103, audio objects in each of the categories are assigned to at least one of the clusters based on the assignment.

[0058] In the following description, two approaches are provided for clustering audio objects after they are classified into categories in S101 and each category is assigned a cluster in S102.

[0059] In one approach, audio objects in each category may be assigned to at least one of the clusters assigned to one or more of those categories based on reducing the distortion cost associated with those categories. That is, due to the limitations on the number of clusters assigned to each category, some leakage across clusters and categories is allowed to reduce the distortion cost and avoid artifacts for complex audio content. This approach may be referred to as fuzzy category clustering. In this fuzzy category clustering approach, an audio object may be softly partitioned with the gains and corresponding costs into different clusters in different categories. During the clustering process, the distortion cost is expected to be minimal with respect to the overall spatial distortion as well as the penalty or inconsistency of allocating objects within a category to clusters of different categories. Thus, there is a trade-off between the cluster budget and the complexity of the audio content. The fuzzy category clustering approach may be suitable for audio objects with metadata such as zone masks and snaps, for which strict separation with other metadata is not required. The fuzzy category clustering approach may be described as follows:

[0060] In the fuzzy category clustering approach, the number of clusters assigned to each category may be determined in S102 based on the importance of the audio objects or based on minimizing the overall spatial distortion. For importance-based cluster assignment, there may be some categories to which no clusters are assigned. In these cases, the fuzzy category clustering approach may be applied when clustering the audio objects, since the objects may be soft-clustered into a cluster or clusters of other categories. There may not be a necessary correlation between the approach applied in the cluster assignment stage and the approach applied in the audio object clustering stage.

[0061] In the fuzzy categorical clustering approach, the distortion cost is calculated by: (1) the original spatial location of each audio object

number

number

number

number

[0062] In some embodiments, the gain g o,m teeth,

number

number

number

[0063]

number

number

[0064] In some embodiments, the cost function is:

number

[0065] The cost function may typically be minimized under some additional criterion. When allocating the audio signals, one criterion may be to preserve the summed amplitude or energy of the input audio objects. For example,

number

[0066] In the following, we may discuss the cost function E. By minimizing the cost function, the gain g o,m can be determined.

[0067] The cost function, as described above, is

number

number

[0068] The cost function is

number

number

number

number

number

number

number

number

number

number

number

[0069] Based on these four terms in the cost function, the payoff g o,m The gain g can be determined. o,m An example of the calculation for is given below. It should be noted that other methods of calculation are possible.

[0070] The gain g of the oth audio object for each M clusters o,m can also be written as a vector:

number

number

number

number

number

[0071] Audio Object o and m The second term E represents the discrepancy between C may be reformulated as follows:

number

[0072] The third term E represents the sum of the gains of the audio objects and a deviation of +1. N may be reformulated as follows:

number

[0073] The fourth term E represents the distance between the original spatial location of the audio object and the reconstructed spatial location. P may be reformulated as follows:

number

number

number

number

number

[0074] The oth audio object is divided into M clusters, and the gain vector

number

[0075] The reconstructed spatial positions of the audio objects may be obtained by equation (17) when the gain vectors are determined. In this regard, a process of determining gains may also be applied in the cluster assignment based on minimizing the overall spatial distortion as described above to determine the reconstructed spatial positions, and thus the spatial positions of each category.

[0076] It should be noted that a quadratic polynomial is used as an example to determine the minimum in the cost function, many other exponent values ​​may be used in other exemplary embodiments, e.g., 1, 1.5, 3, etc.

[0077] A fuzzy category clustering approach for audio object clustering was described above. In another approach, audio objects in each category may be assigned to at least one of the clusters assigned to that category based on reducing the spatial distortion cost associated with that category. That is, no leakage across categories is allowed. Audio object clustering is performed within each category, and audio objects may not be grouped into clusters assigned to another category. This approach may be referred to as a hard category clustering approach. In some embodiments where this approach is applied, an audio object may be assigned to two or more of the clusters assigned to the category corresponding to that audio object. In a further embodiment, no leakage across clusters is allowed, and an audio object may be assigned to only one of the clusters assigned to the corresponding category.

[0078] The hard category clustering approach may be suitable for some specific applications, such as dialogue replacement or dialogue enhancement, that require audio objects (dialogue objects) to be separated from others.

[0079] In a hard category clustering approach, audio objects in a category cannot be clustered into one or more clusters in other categories, so it is expected that at least one cluster has been assigned to each category in the previous cluster assignment. To this end, in some embodiments, cluster assignment by minimizing the overall spatial distortion as described above may be more suitable. In other embodiments, importance-based cluster assignment may also be used when hard category clustering is applied. Some additional conditions may be used in the cluster assignment to ensure that at least one cluster is assigned to each category as discussed above. For example, a minimum threshold of clusters or a minimum threshold of spatial distortion for each category may be utilized.

[0080] Within a category, audio objects may be clustered in only one cluster or in multiple clusters in one or more exemplary embodiments, since the categories represent the same type of metadata. For example, as shown in FIG. 2, audio objects in category 1 may be clustered into one or more of clusters 4, 5 or 6. In a scenario where an audio object is clustered into multiple clusters within a category, the corresponding gain may also be determined to reduce or even minimize the distortion cost associated with that category (this may be similar to what was mentioned with respect to the fuzzy category clustering approach). The difference is that the determination is performed within a category. In some embodiments, each input audio object may be allowed to be clustered into only one cluster assigned to its category.

[0081] Two approaches for audio clustering are discussed above. It should be noted that these two approaches can be used separately or in combination. For example, after audio object classification in S101 and cluster assignment in S102, for some of the categories, a fuzzy category clustering approach may be applied to cluster the audio objects within them; for the remaining categories, a hard category clustering approach may be applied. That is, within some categories, some leakage across categories may be acceptable, and for other categories, leakage across categories is not acceptable.

[0082] After the input audio objects are assigned to clusters, for each cluster, the audio objects may be combined to obtain a clustered audio object, and the metadata of the audio objects in each cluster may be combined to obtain the metadata of the clustered audio object. The clustered audio object may be a weighted sum of all audio objects in a cluster using corresponding gains. The metadata of the clustered audio object may in some examples be the corresponding metadata representing its category, or in other examples may be the metadata of any audio object or the most important audio object in the cluster or its category.

[0083] Since all input audio objects are classified into corresponding categories depending on the to-be-preserved information in the metadata before audio object clustering, different to-be-preserved metadata or unique combinations of to-be-preserved metadata are associated with different categories. After clustering, for audio objects in a category, there is less possibility of being mixed with audio objects associated with different metadata. In this regard, the metadata of audio objects can be preserved after clustering. Furthermore, spatial distortion or distortion cost is considered during the cluster assignment and audio object allocation process.

[0084] Fig. 3 illustrates a block diagram of a system 300 for audio object clustering with metadata preservation according to an example embodiment. As illustrated in Fig. 3, the system 300 comprises an audio object classification unit 301 configured to classify a plurality of audio objects into a number of categories based on information to be preserved in metadata associated with the plurality of audio objects. The system 300 further comprises a cluster allocation unit 302 configured to assign a predetermined number of clusters to the categories, and an audio object allocation unit 303 configured to allocate audio objects in each of the categories to at least one of the clusters based on the assignment.

[0085] In some embodiments, the information may include one or more of audio object size information, zone mask information, snapping information, content type, or rendering mode.

[0086] In some embodiments, the audio object classification unit 301 may be further configured to classify audio objects with no information to be stored into one category; and classify audio objects with different information to be stored into a different category.

[0087] In some embodiments, the cluster allocation unit 302 may further comprise: an importance based determination unit configured to determine the number of audio objects from the plurality of audio objects based on an importance of each audio object relative to other audio objects; and a distribution determination unit configured to determine a distribution of the number of audio objects among the categories. In these embodiments, the cluster allocation unit 302 may be further configured to allocate the number of clusters to the categories based on the distribution.

[0088] In some embodiments, the cluster assignment unit 302 may be further configured to assign the predetermined number of clusters to the categories based on reducing an overall spatial distortion for the categories.

[0089] In some embodiments, the overall spatial distortion for the category may comprise a maximum spatial distortion among the individual spatial distortions of the category or a weighted sum of the individual spatial distortions of the category. The spatial distortion for each category may be associated with the original spatial location of each audio object in that category and with the spatial location of at least one of the clusters.

[0090] In some embodiments, a reconstructed spatial position of each audio object may be determined based on the spatial positions of the at least one cluster, and a spatial distortion for each category may be determined based on the distance between the original spatial position of each audio object in that category and the reconstructed spatial position of that audio object.

[0091] In some embodiments, the multiple audio objects may be within a single frame of the audio signal, and the spatial distortion for each category may be further based on a difference between a number of clusters assigned to that category in the current frame and a number of clusters assigned to that category in a previous frame.

[0092] In some embodiments, the cluster assignment unit 302 may be further configured to iteratively reduce the overall spatial distortion for the categories based on at least one of: an amount of spatial distortion for a category in a previous iteration or a difference between the spatial distortion for a category in a current iteration and a previous iteration.

[0093] In some embodiments, the cluster allocation unit 302 may be further configured to allocate the predetermined number of clusters to the categories based on one or more of: a first threshold for the number of clusters assigned to each category, a second threshold for the spatial distortion for each category, or an importance of each category with respect to other categories.

[0094] In some embodiments, the system 300 may further comprise an audio object reclassification unit configured to reclassify at least one audio object in one category to another category based on a spatial distortion for that category.

[0095] In some embodiments, the audio object allocation unit 303 may further allocate the audio objects in each category to at least one of the clusters assigned to that category based on reducing a distortion cost associated with that category.

[0096] In some embodiments, the audio object allocation unit 303 may further allocate the audio objects in each category to at least one of the clusters assigned to one or more of the categories based on reducing the distortion cost associated with those categories.

[0097] In some embodiments, the distortion cost may be associated with one or more of the original spatial location of each audio object, the spatial location of the at least one cluster, an identity of a category into which each audio object is classified, or an identity of each category to which the at least one cluster is assigned.

[0098] In some embodiments, the distortion cost may be determined based on one or more of: a distance between an original spatial location of each audio object and a spatial location of the at least one cluster, a distance between an original spatial location of each audio object and a reconstructed spatial location of that audio object determined based on the spatial location of the at least one cluster, or a mismatch between an identification of a category into which each audio object is classified and an identification of each category to which the at least one cluster is assigned.

[0099] In some embodiments, the system 300 may further comprise an audio object combination unit configured to combine the audio objects in each cluster to obtain a clustered audio object, and a metadata combination unit configured to combine metadata of the audio objects in each cluster to obtain metadata of the clustered audio object.

[0100] For clarity, some additional components of system 300 are not depicted in FIG. 3. However, it should be understood that all features described above with reference to FIG. 1 are applicable to system 300. Furthermore, the components of system 300 may be hardware modules or software unit modules, etc. For example, in some embodiments, system 300 may be implemented partially or completely in software and / or firmware implemented as a computer program product embodied in a computer-readable medium. Alternatively or additionally, system 300 may be implemented partially or completely based on hardware, for example as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SOC), a field programmable gate array (FPGA), etc.

[0101] FIG. 4 illustrates a block diagram of an exemplary computer system 400 suitable for implementing the embodiments. As shown, the computer system 400 has a central processing unit (CPU) 401 that can execute various processes based on a program stored in a read-only memory (ROM) 402 or a program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, data required when the CPU 401 executes various processes and the like is also stored as needed. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0102] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 executes a communication process via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 410 as necessary, and a computer program read from the removable medium 411 is installed in the storage section 408 as necessary.

[0103] Specifically, according to exemplary embodiments disclosed herein, the process described above with reference to Figure 1 may be implemented as a computer software program. For example, an embodiment of the exemplary embodiment includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the method 100. In such an embodiment, the computer program may be downloaded from a network via the communication unit 409, mounted, and / or installed from a removable medium 411.

[0104] In general, various exemplary embodiments may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the exemplary embodiments have been illustrated and described as block diagrams, flowcharts, or using some other pictorial representations, the blocks, devices, systems, techniques, or methods described herein may be implemented in, by way of non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing device, or some combination thereof.

[0105] Additionally, the various blocks illustrated in the flowcharts may be viewed as method steps and / or as operations resulting from computer program code operations and / or as multiple coupled logic circuit elements configured to perform the associated function(s). For example, embodiments include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the methods described above.

[0106] In the context of this disclosure, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] Computer program codes for carrying out the methods of the exemplary embodiments may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, a special purpose computer or other programmable data processing apparatus, such that when the program codes are executed by the processor of the computer or other programmable data processing apparatus, they cause the implementation of the functions / operations specified in the flowcharts and / or block diagrams. The program codes may be executed entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer, partially on a remote computer, or entirely on a remote computer or server. The program codes may also be distributed on specially programmed devices, which may be generally referred to herein as "modules". The software component parts of the modules may be written in any computer language and may be part of a monolithic code base or may be developed in more discrete code portions, as is typical for object-oriented computer languages. Furthermore, the modules may be distributed across multiple computer platforms, servers, terminals, mobile devices, etc. A given module may be implemented such that the functions described are performed by separate processors and / or computing hardware platforms.

[0108] As used herein, the term "circuitry" refers to all of the following: (a) hardware-only circuit implementations (e.g., implemented with only analog and / or digital circuitry) and (b) combinations of circuitry and software (and / or firmware), such as (as appropriate): (i) a combination of processor(s) or (ii) processor(s) / software (including digital signal processors), software and memory(s) that together cause a device such as a cell phone or server to perform various functions, and (c) circuitry that requires software or firmware for operation, even if the software or firmware is not physically present, such as a microprocessor(s) or part of a microprocessor(s). Additionally, those skilled in the art will be familiar with the concept that communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any delivery media.

[0109] Furthermore, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or sequentially, or that all of the shown operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of what may be claimed. Rather, these should be construed as descriptions of features that may be specific to specific exemplary embodiments. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination.

[0110] Various modifications and adaptations to the exemplary embodiments described above may become apparent to those skilled in the art in light of the above description, when the description is read in conjunction with the accompanying drawings. Any and all modifications fall within the scope of the non-limiting exemplary embodiments. Moreover, other exemplary embodiments described herein will occur to those skilled in the art having the benefit of the teachings presented in the above description and drawings.

[0111] Thus, the exemplary embodiments disclosed herein may be embodied in any of the forms described herein. For example, the following enumerated example embodiments describe certain structures, features, and functions of certain aspects of the exemplary embodiments disclosed herein. [EEE1] A method for preserving object metadata in audio object clustering, comprising: allocating audio objects to categories, each category representing one or a unique combination of metadata requiring preservation; and generating a number of clusters for each category through a clustering process, subject to an overall (maximum) number of available clusters and an overall error criterion, the method further comprising: fuzzy object category separation or hard object category separation. [EEE2] The fuzzy object category separation includes the steps of: determining the output cluster centroids, for example by selecting the most important objects; (1) determining the location metadata of each object;

number

number

[0112] It will be understood that the embodiments of the exemplary embodiments disclosed herein are not limited to the particular embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0113] Several aspects will be described. [Aspect 1] 1. A method for audio object clustering in which metadata is preserved, comprising: classifying a plurality of audio objects into categories based on information to be stored in metadata associated with the plurality of audio objects; assigning a predetermined number of clusters to said categories; and allocating audio objects in each of the categories to at least one of the clusters based on the assignments. method. [Aspect 2] 2. The method of embodiment 1, wherein the information includes one or more of audio object size information, zone mask information, snap information, content type, or rendering mode. [Aspect 3] Classifying a plurality of audio objects into categories based on information to be stored in metadata associated with the plurality of audio objects, comprising: Classify audio objects with no information to be preserved into one category; This involves classifying audio objects into different categories, each with different information to be preserved. The method of embodiment 1. Aspect 4 The step of assigning a predetermined number of clusters to the categories comprises: determining the number of audio objects from the plurality of audio objects based on an importance of each audio object relative to the other audio objects; determining a distribution of said predetermined number of audio objects among said categories; assigning the predetermined number of clusters to the categories based on the distribution. The method of embodiment 1. Aspect 5 The step of assigning a predetermined number of clusters to the categories comprises: assigning the number of clusters to the categories based on reducing an overall spatial distortion for the categories. The method of embodiment 1. Aspect 6 the overall spatial distortion for the category comprises a maximum spatial distortion among the individual spatial distortions of the category or a weighted sum of the individual spatial distortions of the category; the spatial distortion for each category being related to the original spatial location of each audio object in that category and to the spatial location of at least one of said clusters; The method of embodiment 5. Aspect 7 The method of embodiment 6, wherein a reconstructed spatial position of each audio object is determined based on the spatial position of the at least one cluster, and a spatial distortion for each category is determined based on the distance between the original spatial position of each audio object in that category and the reconstructed spatial position of that audio object. Aspect 8 The method of embodiment 6, wherein the multiple audio objects are within one frame of the audio signal, and the spatial location for each category is further based on a difference between the number of clusters assigned to that category in the current frame and the previous frame. Aspect 9 Allocating the predetermined number of clusters to the categories based on reducing overall spatial distortion for the categories comprises: The amount of spatial distortion for a category in the previous iteration, or The difference between the spatial distortion for a category in the current iteration and the previous iteration and iteratively reducing an overall spatial distortion for the category based on at least one of The method of embodiment 5. Aspect 10 The step of assigning a predetermined number of clusters to said categories further comprises: a first threshold for the number of clusters to be assigned to each category; A second threshold for spatial distortion for each category or The importance of each category relative to the others 10. The method of any one of embodiments 4 to 9, based on one or more of the following: Aspect 11 reclassifying at least one audio object in one category into another category based on the spatial distortion for that category. The method of embodiment 1. Aspect 12 allocating audio objects in each of the categories to at least one of the clusters based on the assignments, comprising: allocating audio objects in each category to at least one of the clusters assigned to that category based on reducing a distortion cost associated with that category. The method of embodiment 1. Aspect 13 allocating audio objects in each of the categories to at least one of the clusters based on the assignments, comprising: allocating audio objects in each category to at least one of the clusters assigned to one or more of the categories based on reducing a distortion cost associated with the category. The method of embodiment 1. Aspect 14 The method of aspect 12 or 13, wherein the distortion cost is related to one or more of the original spatial location of each audio object, the spatial location of the at least one cluster, an identification of a category into which each audio object is classified, or an identification of each category to which the at least one cluster is assigned. Aspect 15 wherein the distortion cost is: a distance between the original spatial location of each audio object and the spatial location of said at least one cluster; a distance between the original spatial location of each audio object and a reconstructed spatial location of that audio object determined based on the spatial locations of the at least one cluster; or a mismatch between the identity of a category into which each audio object is classified and the identity of each category to which said at least one cluster is assigned; The method of embodiment 14, wherein the determination is based on one or more of: Aspect 16 combining the audio objects in each cluster to obtain clustered audio objects; and combining metadata of the audio objects in each cluster to obtain metadata of the clustered audio objects. The method of embodiment 1. Aspect 17 1. A system for audio object clustering in which metadata is preserved, comprising: an audio object classification unit configured to classify a plurality of audio objects into categories based on information to be stored in metadata associated with the plurality of audio objects; a cluster allocation unit configured to allocate a predetermined number of clusters to said categories; an audio object allocation unit configured to allocate audio objects in each of the categories to at least one of the clusters based on the allocation; system. Aspect 18 20. The system of claim 17, wherein the information includes one or more of audio object size information, zone mask information, snap information, content type, or rendering mode. Aspect 19 The system of embodiment 17, wherein the audio object classification unit is further configured to classify audio objects without information to be stored into one category and classify audio objects with different information to be stored into a different category. Aspect 20 The cluster allocation unit: an importance based decision unit configured to decide the number of audio objects from the plurality of audio objects based on an importance of each audio object relative to other audio objects; a distribution determination unit configured to determine a distribution of the number of audio objects among the categories, the cluster allocation unit is further configured to allocate the predetermined number of clusters to the categories based on the distribution. 18. The system of embodiment 17. Aspect 21 20. The system of claim 17, wherein the cluster assignment unit is further configured to assign the predetermined number of clusters to the categories based on reducing an overall spatial distortion for the categories. Aspect 22 the overall spatial distortion for the category comprises a maximum spatial distortion among the individual spatial distortions of the category or a weighted sum of the individual spatial distortions of the category; the spatial distortion for each category being related to the original spatial location of each audio object in that category and to the spatial location of at least one of said clusters; 22. The system of embodiment 21. Aspect 23 The system of embodiment 22, wherein a reconstructed spatial position of each audio object is determined based on the spatial position of the at least one cluster, and a spatial distortion for each category is determined based on the distance between the original spatial position of each audio object in that category and the reconstructed spatial position of that audio object. Aspect 24 23. The system of claim 22, wherein the multiple audio objects are within one frame of the audio signal, and the spatial location for each category is further based on the difference between the number of clusters assigned to that category in the current frame and the previous frame. Aspect 25 The cluster allocation unit further comprises: The amount of spatial distortion for a category in the previous iteration, or The difference between the spatial distortion for a category in the current iteration and the previous iteration and iteratively reducing an overall spatial distortion for the category based on at least one of 22. The system of embodiment 21. Aspect 26 The cluster allocation unit further comprises: allocating the predetermined number of clusters to the categories: a first threshold for the number of clusters to be assigned to each category; A second threshold for spatial distortion for each category or The importance of each category relative to the others 26. The system of any one of aspects 20 to 25, configured to perform based on one or more of the following: Aspect 27 an audio object reclassification unit configured to reclassify at least one audio object in one category to another category based on a spatial distortion for the category; 18. The system of embodiment 17. Aspect 28 The system of embodiment 17, wherein the audio object allocation unit is further configured to allocate audio objects in each category to at least one of the clusters assigned to that category based on reducing a distortion cost associated with that category. Aspect 29 The system of aspect 17, wherein the audio object allocation unit is further configured to allocate audio objects in each category to at least one of the clusters assigned to one or more of the categories based on reducing a distortion cost associated with the category. Aspect 30 The system of embodiment 28 or 29, wherein the distortion cost is related to one or more of the original spatial location of each audio object, the spatial location of the at least one cluster, an identification of a category into which each audio object is classified, or an identification of each category to which the at least one cluster is assigned. Aspect 31 wherein the distortion cost is: a distance between the original spatial location of each audio object and the spatial location of said at least one cluster; a distance between the original spatial location of each audio object and a reconstructed spatial location of that audio object determined based on the spatial locations of the at least one cluster; or a mismatch between the identity of a category into which each audio object is classified and the identity of each category to which said at least one cluster is assigned; The system of embodiment 30, wherein the determination is based on one or more of: Aspect 32 an audio object combination unit configured to combine the audio objects in each cluster to obtain a clustered audio object; a metadata combination unit configured to combine metadata of the audio objects in each cluster to obtain metadata of the clustered audio objects. 18. The system of embodiment 17. Aspect 33 17. A computer program product having embodied on a machine-readable medium a computer program comprising program code for performing the method of any one of aspects 1 to 16.

Claims

1. 1. A method of decoding an encoded audio signal, comprising: receiving the encoded audio signal and determining at least one audio object from the encoded audio signal; receiving zone metadata indicating areas in which the audio object should not be rendered; classifying the at least one audio object into at least one category based on rendering mode metadata associated with the at least one audio object; determining at least one cluster for each category; rendering the at least one audio object based on the rendering mode metadata for the at least one cluster, the rendering further based on the zone metadata. method.

2. A system for decoding an encoded audio signal, comprising: a receiver for receiving the encoded audio signal and determining at least one audio object from the encoded audio signal, the receiver being further configured to receive zone metadata indicating areas in which the audio object should not be rendered; an audio object classification unit for classifying the at least one audio object into at least one category based on rendering mode metadata associated with the at least one audio object; a cluster assignment unit for determining at least one cluster for each category; a renderer for rendering the at least one audio object based on the rendering mode metadata for the at least one cluster, the rendering further based on the zone metadata. system.

3. A computer program for causing a computer to execute the method of claim 1.

Images