Metadata-preserving audio object clustering

Abstract

Embodiments of the present invention relate to metadata-preserving audio object clustering. A method for metadata-preserving audio object clustering is disclosed. The method includes classifying a plurality of audio objects into a plurality of categories based on information to be preserved in metadata associated with the plurality of audio objects. The method also includes assigning a predetermined number of clusters to the categories, and allocating audio objects in each of the categories to at least one of the clusters based on the assignment. A corresponding system and computer program product are also disclosed.

Description

[0001] This application is a divisional application of the invention patent application with application date of December 11, 2014, application number 201410765578.6, and invention name “Clustering of Audio Objects with Metadata Preservation”. Technical Field

[0002] The present invention relates generally to audio content processing and, more particularly, to a method and system for clustering audio objects that allows metadata to be preserved. Background Art

[0003] The advent of object-based audio has significantly increased the amount of audio data and the complexity of rendering that data in high-end playback systems. For example, a cinema soundtrack may include many different sound elements corresponding to images, dialogue, noise, and sound effects on the screen that originate from different locations on the screen and are combined with background music and ambient effects to create an overall auditory experience. Accurate playback requires that the sound should be reproduced in a way that corresponds as closely as possible to what is displayed on the screen in terms of source location, intensity, motion, and depth. Object-based audio represents a significant improvement over traditional channel-based audio systems, which send audio content to a single speaker in a listening environment in the form of speaker feeds and are therefore relatively limited in terms of spatial playback of specific audio objects.

[0004] The introduction of digital cinema and the development of three-dimensional ("3D") content has created new sound standards, such as the incorporation of multiple channels of audio to allow greater creativity for content creators and a more enveloping and realistic auditory experience for listeners. Expanding beyond traditional speaker feeds and channel-based audio as a means for distributing spatial audio is critical, and a focus has been on model-based audio descriptions that allow listeners to select a desired playback configuration with audio rendered specifically for their selected configuration. Spatial representations of sound utilize audio objects, which are audio signals with associated parametric source descriptions of apparent source positions (e.g., 3D coordinates), apparent source widths, and other parameters. Further improvements include the development of next generation spatial audio (also known as "adaptive audio") formats that include a mix of audio objects and traditional channel-based speaker feeds (audio substrates) along with positional metadata for the audio objects.

[0005] As used herein, the term "audio object" refers to an individual audio element that exists in a sound field for a certain period of time. The term "audio floor" or "floor" refers to the audio channels to be reproduced in predefined and fixed speaker positions.

[0006] In some soundtracks, there may be several (e.g., 7, 9, or 11) bed channels containing audio. In addition, based on the capabilities of the authoring system, there may be dozens or even hundreds of individual audio objects that are combined during rendering to create a spatially diverse and immersive audio experience. In other distribution and transmission systems, there may be sufficient available bandwidth to transmit all audio bed channels and objects with little or no audio compression. However, in some cases, such as Blu-ray Disc, broadcast (cable, satellite, and terrestrial), mobile (3G and 4G), and over-the-top (OTT or Internet) distribution, there may be significant limitations on the available bandwidth for digitally transmitting all bed channel and object information created at the time of authoring. Although audio encoding methods (lossy or lossless) can be applied to audio to reduce the required bandwidth, audio encoding may not be sufficient to reduce the bandwidth required to transmit audio, especially over very limited networks such as mobile 3G and 4G networks.

[0007] Some existing methods have been developed to reduce the number of input objects to a smaller set of output objects by means of clustering. Generally speaking, in some clustering processes, metadata such as size, zone mask and snap should be pre-rendered to the internal channel structure. The clustering of audio objects only aggregates the spatial positions of the audio objects, and the output objects contain only spatial metadata. Such output objects may not work well for some reproduction systems because the loss of metadata may violate the desired artistic intent.

[0008] The subject matter discussed in the background section should not be assumed to be prior art simply because it is disclosed in the background section. Similarly, problems mentioned in the background section or problems related to the subject matter of the background section should not be assumed to be known in advance in the prior art. The subject matter in the background section only presents different methods, which themselves may also be inventions. Summary of the invention

[0009] To solve the above problems, example embodiments propose a method and system for audio object clustering with metadata preservation.

[0010] In one aspect, an example embodiment provides a method for clustering audio objects for metadata preservation. The method includes classifying a plurality of audio objects into a plurality of categories based on information to be preserved in metadata associated with the plurality of audio objects. The method also includes assigning a predetermined number of clusters to the categories, and allocating audio objects in each of the categories to at least one of the clusters based on the assignment. Embodiments of this aspect also include corresponding computer program products.

[0011] In another aspect, an example embodiment provides a system for audio object clustering for metadata preservation. The system includes an audio object classification unit configured to classify a plurality of audio objects into a plurality of categories based on information to be preserved in metadata associated with the plurality of audio objects. The system also includes a cluster assignment unit configured to assign a predetermined number of clusters to the categories, and an audio object allocation unit configured to assign audio objects in each of the categories to at least one of the clusters according to the assignment.

[0012] It will be understood from the following description that according to an embodiment of the present invention, input audio objects are classified into corresponding categories based on the information to be retained in their metadata, so that different metadata to be retained or specific combinations of metadata to be retained are associated with different categories. After clustering, it is less likely that audio objects in one category will be mixed with audio objects associated with different metadata. In this regard, the metadata of the audio objects can be retained after clustering. Other benefits brought by embodiments of the present invention will be clear from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown by way of example and not limitation, wherein:

[0014] Figure 1 A flowchart showing a method for audio object clustering with metadata preservation according to an example embodiment of the present invention is shown;

[0015] Figure 2 A schematic diagram of a process for clustering audio objects according to an example embodiment of the present invention is shown;

[0016] Figure 3 A block diagram showing a system for metadata-preserving audio object clustering according to an example embodiment of the present invention; and

[0017] Figure 4 A block diagram of a computer system suitable for implementing an example embodiment of the present invention is shown.

[0018] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0019] The principle of the present invention will be described below with reference to several exemplary embodiments shown in the accompanying drawings. It should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0020] As mentioned above, due to the limitations of encoding / decoding rate and transmission bandwidth, the number of audio objects used to generate adaptive audio content should be reduced by means of clustering. In addition to the metadata describing the spatial position of the audio object, the audio object usually has other metadata describing its attributes, such as size, area masking, capture, and content type, each of which describes the creative intent of how the audio object should be processed when rendered. However, in some existing methods, after the audio object is clustered, only the position metadata is retained. Although other metadata can be pre-rendered to the internal channel architecture, such as in a 7.1.2 or 7.1.4 system, this method does not work well for all systems. In particular, when the audio object is down-mixed to, for example, a 5.1 or 7.1 system, the creative intent of the audio object may be violated when rendering.

[0021] Taking the metadata "region masking" as an example, it has multiple modes and each mode defines an area where an audio object should not be rendered. One mode of region masking is "no sides", which describes that the side speakers should be masked when rendering the audio object. By using traditional clustering methods, if an audio object at spatial position z=1 is rendered to a 5.1 system with metadata "no sides", the side speakers may be activated in the 5.1 rendering because the sound at the ceiling speakers will be folded to the sides. This violates the original creative intention. To solve this problem, the metadata "region masking" should be retained during the clustering process so that it can be processed correctly at the audio renderer.

[0022] In another example, it may be desirable to separate the dialog object from other objects after clustering, which may have many benefits for subsequent audio object processing. For example, in subsequent audio processing such as dialog enhancement, the separated dialog object cluster may be enhanced by simply applying one or gain. Otherwise, if the dialog object is mixed with other objects in a cluster, it may be difficult to separate the dialog object. In the application of dialog replacement, the dialogs in each language may be completely separated from each other. For these purposes, in the clustering process, the dialog object should be retained and assigned to a separate specific cluster.

[0023] Furthermore, an audio object may be associated with metadata describing its rendering mode, e.g., as left total / right total (Lt / Rt) or as binaural with head related transfer function (HRTF) when processed in a headphone renderer. It is also desirable that these rendering modes be preserved after clustering in order to generate good rendering results.

[0024] Therefore, in order to achieve a better audio experience, it is desirable to retain metadata in audio object clustering. Embodiments of the present invention provide a method and system for audio object clustering with metadata retention.

[0025] First refer to Figure 1 , which describes a method 100 for metadata-preserving audio object clustering according to an example embodiment of the present invention.

[0026] At S101, multiple audio objects are classified into multiple categories based on information to be retained in metadata associated with the multiple audio objects. Audio objects are provided as input, and there may be tens, hundreds, or sometimes thousands of input audio objects.

[0027] As used herein, the information to be retained in the metadata associated with each audio object can indicate the processing intention when the audio object is rendered. This information can describe how the audio object should be processed when it is rendered. In some embodiments, this information can include at least one or more of the following: size information, regional masking information, capture information, content type, and rendering mode of the audio object. The size information can be used to indicate the spatial area or volume occupied by the audio object. The regional masking information indicates the mode of regional masking, which defines the area where the audio object should not be rendered. For example, the regional masking information can indicate a "no side" mode, a "surround only" mode, a "front only" mode, etc. The capture information indicates whether the audio object should be directly panned to the nearest speaker.

[0028] It should be noted that only some examples of information to be retained in metadata are described, and other information retained in metadata (such non-limiting examples include spatial position, spatial width, etc.) may also be retained in audio object classification according to the preferences of the user or audio engineer. In some embodiments, all information in the metadata associated with the audio object may be considered.

[0029] The number of categories may depend on the information in the metadata of the audio object and may be greater than or equal to 1. In one embodiment, audio objects without information to be retained may be classified into one category, and audio objects with different information to be retained may be classified into different categories. That is, depending on the different information to be retained, the corresponding audio objects are classified into different categories. Alternatively, the categories may represent specific combinations of different information to be retained in the metadata. All other audio objects that do not have information of interest may be included in one category, and in some cases may be included in multiple categories. The scope of the present invention is not limited in this respect.

[0030] These categories can be given by manual assignment, automatic assignment or a combination of the two. For example, a user or an audio engineer can mark the audio objects associated with different types of metadata by different signs, and then these marked audio objects can be classified into different categories according to their signs. For another example, the information to be retained in the metadata can be automatically identified. The user or audio engineer can also pre-configure their preferences or expectations, such as classifying dialogue objects, separating different dialogue languages, and / or classifying different regional masking modes. According to the pre-configuration, audio objects can be classified into different categories.

[0031] Assume there are O audio objects. During the classification process, the information to be retained in the metadata of the audio objects can be obtained based on: (1) metadata labels provided by user input, such as region masking labels, or crawling labels, or audio type labels, or language labels, and / or (2) automatic identification / labeling of metadata, such as but not limited to, identification of content type. The number of possible categories N can be determined based on the information obtained, each category consisting of a specific combination of information to be retained. After classification, each audio object can have an associated category identifier n o .

[0032] Reference Figure 2 , which shows a schematic diagram of audio object clustering. Figure 2 As shown, based on the information to be retained in the metadata, multiple input audio objects are classified into five categories, categories 0 to 4. An example of these categories can be given as follows:

[0033] Category 0: all audio objects with no information to be preserved;

[0034] Category 1: music object, no region masking;

[0035] Category 2: Sound effect objects with region masking "surround only";

[0036] Category 3: English conversation partners; and

[0037] Category 4: Spanish dialogue object with region mask "frontend only".

[0038] The input audio object may include one or more frames. A frame is a processing unit of audio content, and the duration of a frame may vary and may depend on the configuration of the audio processing system. Because the audio objects to be classified may vary for different frames in time and their metadata may also vary, the number of categories may also vary over time. The categories representing different types of information to be retained may be pre-defined by the user or defaulted, and the input audio objects in one or more frames may then be classified into predetermined categories based on the information. In subsequent processing, categories with classified audio objects may be considered and those categories without audio objects may be ignored. For example, in Figure 2 In the example, when there is no audio object without information to be retained, the corresponding category 0 may be omitted. It is conceivable that the number of audio objects classified in each category may vary over time.

[0039] At S102, a predetermined number of clusters are assigned to the category. The predetermined number may be greater than 1 and may depend on the transmission bandwidth and encoding / decoding rate of the audio processing system. There may be a trade-off between the transmission bandwidth (and / or encoding rate, and / or decoding rate) and the error criterion of the output audio object. For example, the predetermined number may be 11 or 16. Other values ​​may also be determined, such as 5, 7, or 20, and the scope of the present invention is not limited in this respect.

[0040] In some embodiments, the predetermined number may not change in the same processing system. In some other embodiments, the predetermined number may change for different audio files to be processed.

[0041] In an embodiment of the present invention, audio objects are first classified into categories according to metadata at S101, so that each category can represent different information to be retained or a specific combination of different information to be retained. However, the audio objects in these categories can be clustered in subsequent processing. There may be many ways to assign / allocate a total predetermined number of clusters to these categories. In some example embodiments, since the total number of clusters is predetermined and fixed, the number of clusters assigned to each category may be determined before clustering the audio objects. Some example embodiments are now discussed.

[0042] In one embodiment, cluster assignment may depend on the importance of multiple audio objects. Specifically, a predetermined number of audio objects may first be determined from the multiple audio objects based on the importance of each audio object relative to other audio objects, and then the distribution of the predetermined number of audio objects among the categories may be determined. Thus, a predetermined number of clusters are assigned to the categories according to the distribution.

[0043] The importance of each audio object can be related to one or more of the following: the content type of the audio object, the relative loudness (partial loudness) level, and the energy level. An audio object with high importance can represent that the audio object is perceptually significant among the input audio objects, for example because of its relative loudness or energy level. In some use cases, one or more content types can be considered important, and thus high importance is given to the corresponding audio object. For example, a larger importance can be assigned to a dialog object. It should be noted that there are many other ways to determine or define the importance of each audio object. For example, the importance level of some audio objects can be specified by the user. The scope of the present invention is not limited in this respect.

[0044] Assume that the predetermined total number of clusters is M. In the first step, the most important M audio objects among the input audio objects are selected. Because all input audio objects are classified into corresponding categories in S101, in the second step, the distribution of the M most important audio objects among the categories can be determined. Based on how many of the M audio objects are distributed in a category, a corresponding number of clusters can be assigned to the category.

[0045] For example, refer to Figure 2 , 11 most important audio objects (illustrated as circle 201) are determined from a plurality of input audio objects (illustrated as a set of circles 201 and 202). After all input audio objects are classified into five categories (category 0 to 4), 11 most important audio objects (illustrated as circle 201) are determined from a plurality of input audio objects (illustrated as a set of circles 201 and 202). Figure 2 It can be seen that the four most important audio objects are classified into category 0, the three most important audio objects are classified into category 1, the one most important audio object is classified into category 2, the two most important audio objects are classified into category 3, and the one most important audio object is classified into category 4. Therefore, 4 clusters, 3 clusters, 1 cluster, 2 clusters, and 1 cluster are assigned to categories 0 to 4, respectively, as shown in FIG. Figure 2 shown.

[0046] It should be noted that the examples of importance criteria described above according to the exemplary embodiments of the present invention may not be so strict. That is, it is not necessary to select the most important audio objects. In some embodiments, an importance threshold may be configured. Among the audio objects whose importance is greater than the threshold, a predetermined number of audio objects may be randomly selected.

[0047] In addition to the importance criterion, cluster assignment can be based on the overall spatial distortion of the categories. That is, a predetermined number of clusters are assigned to the categories based on reducing or even minimizing the overall spatial distortion of the categories.

[0048] In one embodiment, the overall spatial distortion of a category may include a weighted sum of the individual spatial distortions of the category. The weight of the corresponding category may represent the importance of the category or the importance of the information to be retained associated with the category. For example, categories with higher importance may have higher weights. In another embodiment, the overall spatial distortion of a category may include the maximum spatial distortion among the individual spatial distortions of the category. It should be appreciated that it is not necessary to select only the maximum value, and in some embodiments, other spatial distortions among these categories, such as the second largest spatial distortion, or the third largest spatial distortion, etc., may be considered as overall spatial distortion.

[0049] The spatial distortion of each category can be represented by the distortion level of the audio objects included in the category, and the distortion level of each audio object can be measured by the distance between its original spatial position and its position after being clustered. In general, the position of the audio object after clustering depends on the spatial position of the (multiple) clusters to which it is assigned. In this case, the spatial distortion of each category is associated with the original spatial position of each audio object in the category and the spatial position of the (multiple) clusters. The original spatial position of the audio object can be included in the metadata of the audio object and, for example, includes three Cartesian coordinates (or similarly, for example, includes polar coordinates, cylindrical and spherical coordinates, homogeneous coordinates, line coordinates, etc.). In one embodiment, in order to calculate the spatial distortion of each category, the reconstructed spatial position of each audio object in the category can be determined based on the spatial position of the (multiple) clusters. However, the spatial distortion of each category can be calculated based on the distance between the original spatial position of each audio object in the category and the reconstructed spatial position of the audio object. The reconstructed spatial position of the audio object is the spatial position of the audio object represented by one or more corresponding spatial clusters. An example method for determining the reconstructed spatial position will be described below.

[0050] In order to obtain the overall spatial distortion, the spatial distortion of each category can first be calculated with respect to different numbers of clusters. There are many methods to determine the spatial distortion of a category of an audio object. One method is given below as an example. It should be noted that other existing methods can be applied to measure the spatial distortion of an audio object (and thus calculate the spatial distortion of a category).

[0051] Assume that for class n, there exist M n A cluster centroid, represented by a spatial location {C n (1),C n (2),...,C n (M n )}. dis(o n (i),{C n (1),Cn (2),...,C n (M n )}) can be used to represent the audio object o n (i) Clustering to M n The spatial distortion of the audio object when the cluster center is n (in this case, it is assumed that the audio objects in a category are only assigned to the cluster associated with the category). The spatial distortion of category n can be expressed as:

[0052]

[0053] Among them, n represents the number of audio objects in category n, and o n (i) represents the i-th audio object in category n. In one embodiment, C n (m) may be the spatial position of the audio object with the mth largest importance in the category, and C n The spatial position of (m) may be the spatial position of the audio object. n (i),{C n (1),C n (2),...,C n (M n )}) can be used by each audio object o n (i) Spatial location The audio object is clustered into M n The reconstructed spatial position of clusters The distance between them (or the square of the distance) is used to determine.

[0054] After obtaining the spatial distortion of each category, in one embodiment, as mentioned above, the overall spatial distortion of these categories can be determined as a weighted sum of the individual spatial distortions. For example, the overall spatial distortion can be determined as follows:

[0055]

[0056] Where N represents the total number of categories. The gain a for each category n It may be predetermined and may indicate the importance of the corresponding category or the importance of the information to be retained in the metadata associated with the category.

[0057] In another embodiment, the overall spatial distortion of a category may be determined as the maximum spatial distortion between the individual spatial distortions of the category. For example, the overall spatial distortion may be determined as follows:

[0058] Cost=max(a0D0(M0),a1D1(M1),...,a n Dn (M n )) (3)

[0059] In this way, the number of clusters M assigned to each class is n It can be determined based on reducing or minimizing the overall spatial distortion metric, with the constraint That is, the total number of assigned clusters is equal to the predetermined number M.

[0060] The input audio object is usually in units of one frame of audio signal. Due to the typical dynamic characteristics of the audio signal, and assuming that the number of audio objects changes in each category, the number of clusters assigned to each category may usually change over time. Since the number of clusters changed in each category may cause some instability problems, the cost metric is used to utilize the modified spatial distortion that takes into account the consistency of the number of clusters. Therefore, the cost metric can be defined as a time function. Specifically, the spatial distortion of each category is further based on the difference between the number of clusters assigned to the category in the current frame and the number of clusters assigned to the category in the previous frame. In this regard, the overall spatial distortion in formula (2) can be modified as follows:

[0061]

[0062] The overall spatial distortion in formula (3) can be modified as follows:

[0063] Cost=max(a0f(D0(M0),M0,M0'),a1f(D1(M1),M1,M1'),...,a n f(D n (M n ),M n ,M n '))(5)

[0064] In formulas (4) and (5), M n represents the number of clusters of category n in the current frame, M n ' represents the number of clusters of category n in the previous frame, and f(D n (M n ),M n ,M n ') represents the modified overall spatial distortion.

[0065] If the number of clusters assigned to a class changes in the current frame, the modified spatial distortion may be increased compared to the previous spatial distortion to prevent the change in the number of clusters. n (M n ),M n ,M n') can be determined as follows:

[0066] f(D n (M n ),M n ,M n ')=D n (M n )+β1*|M n -M n '| (6)

[0067] Where β1 denotes a parameter with a positive value. According to the modified spatial distortion, there is a penalty for the change of the number of clusters for each category. Therefore, the spatial instability introduced by the transformation of the number of clusters can be alleviated.

[0068] Since a reduction in the number of clusters of a class is more likely to introduce spatial instability than an increase in the number of clusters, in another embodiment, f(D n (M n ),M n ,M n ') can be determined as follows:

[0069]

[0070] Where β2 represents a parameter with a value greater than 1. In this embodiment, there is a larger penalty for reducing the number of clusters, especially when the spatial distortion of the class with the reduced number of clusters is larger. Therefore, the spatial instability introduced by the reduced number of clusters can be reduced.

[0071] In the above description, regarding cluster assignment based on reducing overall spatial distortion, a large amount of computational work may be involved in determining the optimal number of clusters for each category. In order to effectively determine the number of clusters for each category, in one embodiment, an iterative process is proposed. That is, by maximizing the cost reduction in each iteration of the cluster assignment process, the optimal number of clusters for each category is estimated, so that the overall spatial distortion of the category can be iteratively reduced or even minimized.

[0072] By iterating from 1 to a predetermined number of clusters M, in each iteration, one or more clusters are assigned to the category that needs them most. Cost(m-1) and Cost(m) are expressed as the overall spatial distortion in the (m-1)th iteration and the mth iteration. In the mth iteration, one or more new clusters can be assigned to the category n* that can best reduce the overall spatial distortion. Therefore, n* can be determined by increasing or maximizing the reduction of the overall spatial distortion, which can be expressed as follows:

[0073]

[0074] The iterative process may be based on the difference between the spatial distortion of a class in the current iteration and the spatial distortion of the class in the previous iteration, or the amount of spatial distortion of a class in the previous iteration.

[0075] For the overall spatial distortion obtained by the weighted sum of all spatial distortions of the category, the iterative process can be based on the difference between the spatial distortion of a category in the current iteration and the spatial distortion of the category in the previous iteration. In each iteration, at least one cluster can be assigned to the following category, the spatial distortion of which becomes sufficiently lower in the current iteration than its spatial distortion in the previous iteration (according to a first predetermined level) if the category is assigned with the at least one cluster. In one embodiment, at least one cluster can be assigned to the category with the greatest degree of spatial distortion reduction when the at least one cluster is assigned. For example, in this embodiment, n* can be determined as follows:

[0076]

[0077] in and represents the number and spatial distortion of clusters of class n* after the (m-1)th iteration. represents the number of clusters of class n* in the mth iteration if a new cluster is assigned / added to class n* in that iteration, and represents the spatial distortion of class n* in the mth iteration. It should be noted that in each iteration, more than one new cluster may be assigned and class n* may be determined similarly.

[0078] For the overall spatial distortion determined to be the largest spatial distortion among all categories, the iterative process can be based on the amount of spatial distortion of a category in the previous iteration. In each iteration, at least one cluster can be assigned to the category that had a spatial distortion above a second predetermined level in the previous iteration. In one embodiment, at least one cluster can be assigned to the category that had the largest spatial distortion in the previous iteration. For example, in this embodiment, n* can be determined as follows:

[0079]

[0080] Since the class with the largest spatial distortion in the previous iteration (if assigned with one or more clusters in the current iteration) can reduce its spatial distortion in the current iteration, the overall spatial distortion determined by the largest spatial distortion among all classes can also be reduced in the current iteration.

[0081] Note that the determinations provided in equations (9) and (10) can be used jointly in an iterative process. For example, in one iteration, equation (9) can be used to assign (multiple) new clusters in that iteration. In another iteration, equation (10) can be used to assign other (multiple) new clusters.

[0082] Two cluster assignment methods have been described above, one based on the importance of audio objects and the other based on reducing overall spatial distortion. Additionally or alternatively, user input can also be used to guide cluster assignment. Because users may have different requirements for different content in different use cases, the flexibility of the clustering process can be greatly improved. In some embodiments, cluster assignment can be further based on one or more of the following: a first threshold for the number of clusters to be assigned to each category, a second threshold for the spatial distortion of each category, and the importance of each category relative to other categories.

[0083] The first threshold can be predefined for the number of clusters to be assigned to each category. The first threshold can be a predetermined minimum number of clusters or a maximum number of clusters for each category. For example, a user can specify that a cluster should have a certain minimum number of clusters. In this case, during assignment, at least the specified number of clusters should be assigned to the category. In the case where a maximum threshold is set, a maximum of a specified number of clusters can be assigned to the category. The second threshold can be set to ensure that the spatial distortion of a category is reduced to a reasonable level. The importance of each category can be specified by the user, or can be determined based on the importance of the audio objects classified into the category.

[0084] In some cases, the spatial distortion of one class may still be high after cluster assignment is completed. To address this issue, in some embodiments, at least one audio object in one class is reclassified to another class based on the spatial distortion of the class. In an example embodiment, if the spatial distortion of one of the classes is above a predetermined threshold, some audio objects in the class may be reclassified to another class until the spatial distortion is less than (or equal to) the threshold. In some examples, audio objects may be reclassified to a class that contains audio objects that have no information to be retained in the metadata, such as Figure 2 In the embodiment where cluster assignment is based on minimizing the overall spatial distortion in an iterative process, object reclassification can also be an iterative process in which the maximum spatial distortion dis(o) is obtained in each iteration. n (i),{C n (1),C n (2),...,C n (M n)})'s audio objects can be reclassified until the spatial distortion criterion for the category is met.

[0085] Due to the typical dynamic properties of audio signals, the importance or spatial position (and therefore spatial distortion) of audio objects changes over time. Therefore, the cluster assignment can be time-varying, and thus the number of clusters assigned to each category can change over time. In this regard, the category identification associated with cluster m can change over time. Specifically, cluster m can represent a certain language (e.g., Spanish) during a first frame, while in a second frame the cluster can change category identification and therefore change language (e.g., English). This is in contrast to older channel-based systems, in which the language is statically coupled to a certain channel, rather than changing dynamically.

[0086] The cluster assignment at S102 is described above.

[0087] Reference Figure 1 , at S103 , according to the assignment, the audio objects in each of the categories are allocated to at least one of the clusters.

[0088] In the following description, two methods are provided for clustering audio objects after the audio objects are classified into categories at S101 and clusters are assigned at S102 .

[0089] In one method, based on reducing the distortion cost associated with these categories, the audio objects in each category are allocated to at least one cluster in the clusters assigned to one or more categories in these categories. That is, since the number of clusters assigned to each category is limited, some leakage across clusters and across categories is allowed, so as to reduce the distortion cost and avoid the artificial noise (artifact) of complex audio content. This method can be referred to as fuzzy category clustering. In this fuzzy category clustering method, audio objects are softly divided into different clusters in different categories with a certain gain and have corresponding costs. In the clustering process, it is expected that the distortion cost is the minimum with respect to the overall spatial distortion and the disadvantage or mismatch of assigning objects in a category to clusters of different categories. Therefore, there is a trade-off between the complexity of cluster budget and audio content. The fuzzy category clustering method can be suitable for audio objects with metadata such as regional masking and crawling, because for these audio objects, it is not strictly separated for other metadata. The fuzzy category clustering method can be described in the method described below.

[0090] In the fuzzy class clustering method, the number of clusters assigned to each category can be determined at S102 based on the importance of the audio object or based on minimizing the overall spatial distortion. For cluster assignment based on importance, there may be some categories that are not assigned any clusters. In these cases, the fuzzy class clustering method can be applied when clustering audio objects because the objects can be softly clustered to one or more clusters of other categories. It should be noted that there may be no necessary correlation between the method applied in the cluster assignment step and the method applied in the audio object clustering step.

[0091] In the fuzzy class clustering method, the distortion cost can be expressed as a cost function associated with one or more of the following: (1) the original spatial position of each audio object (2) The identifier of the category to which each audio object is classified n o , (3) the spatial location of each cluster, or more specifically, the spatial location of the cluster(s) to which the audio object will be assigned, and (4) the identity of the class associated with each cluster m In one example, the clustered audio objects of a cluster can be obtained by dividing all input audio objects by a gain g o,m The distribution is determined therein and can be expressed as:

[0092]

[0093] Where O represents the number of input audio objects, y m represents the clustered audio object of the mth cluster, x o represents the oth input audio object, and the gain g o,m can be expressed as For example, Figure 2 As shown, audio objects in category 1 can be clustered into all 11 clusters with corresponding gains, regardless of which categories the clusters are assigned to.

[0094] In some embodiments, g o,m The gain can be minimized by n o , and n m The cost function may be based on the original spatial position of each audio object. The spatial position of the cluster to which the audio object is assigned As discussed above, can be determined as the spatial position of the audio object with the greatest importance within the mth category. and The distance between them is as small as possible. Alternatively or additionally, the cost function may also be associated with the identifier n of the category to which each audio object is classified o With the identity of each class to which a cluster is assigned n m Generally speaking, it is expected that audio objects are clustered in the same category so that the cost can be small.

[0095] In some embodiments, the cost function can be expressed as n o , and n m The cumulative contribution of the second-order polynomial, and the global minimum can be determined from the cost function as the gain g o,m A detailed discussion is provided in the manner set forth below.

[0096] The cost function is usually minimized with respect to some additional criterion. When allocating audio objects, one criterion is to maintain the total amplitude or energy of the input audio objects, e.g.

[0097]

[0098] where α can be a value between 1 and 2. For an audio object o, the gain g corresponding to all M clusters o,m It can obey the above equation.

[0099] In the following, the cost function E can be discussed. By minimizing the cost function, the gain g can be determined o,m .

[0100] As mentioned above, the cost function can be expressed as and is associated with the distance between them, which can be regarded as the first term of the cost function E D , and can be determined as:

[0101]

[0102] The cost function can also be compared with n o With n m The mismatch between C . E C The cost of clustering audio objects across clusters within different categories may be represented and may be determined as:

[0103]

[0104] Where n m ! =n0 can be determined as:

[0105]

[0106] As mentioned above, when minimizing the cost function, one criterion is to maintain the summed amplitude or energy of the input audio objects. Therefore, the cost function can also be associated with the gain or loss of energy, that is, the deviation of the gain of a particular audio object from +1. This deviation can be regarded as the third term E of the cost function. N , which can be determined as:

[0107]

[0108] Alternatively, the cost function can be based on the spatial position of each audio object The reconstructed spatial position of the audio object Reconstructing spatial position The gain g can be adjusted according to the audio object o,m The spatial location of the clusters is determined by For example, can be determined as follows:

[0109]

[0110] and The distance between can be considered as the fourth term of the cost function E P , and can be expressed as:

[0111]

[0112] Based on the first, second, third and fourth terms, the cost function can be expressed as a weighted sum of these terms and can be expressed as:

[0113] E=w D E D +w C E C +w N E N +w P E P (19)

[0114] The weight w D , w C , w N and w P It can represent the importance of different items in the cost function.

[0115] Based on the four terms in the cost function, the gain g can be determined o,m Calculate g o,m An example of is given below. It should be noted that other calculation methods are also possible.

[0116] For M clusters, the gain g of the oth object o,m can be written as a vector:

[0117]

[0118] The spatial distortion of M clusters can be written as a matrix:

[0119]

[0120] The matrix of the original spatial position of the audio object can also be constructed as:

[0121]

[0122] The first term E represents the distance between the original spatial position of the audio object and the reconstructed spatial position D can be re-expressed as:

[0123]

[0124] where Λ D Represents a diagonal element The diagonal matrix of .

[0125] n represents the audio object o With n m The second term of the mismatch between C can be re-expressed as:

[0126]

[0127] where Λ C Represents a diagonal element λ mm =(n m ! =n o ) is a diagonal matrix.

[0128] The third term E represents the gain of the audio object and the deviation from +1 N can be re-expressed as:

[0129]

[0130] Among them J N,M represents an all-ones matrix with dimension (N,M).

[0131] The fourth term E represents the distance between the original spatial position of the audio object and the reconstructed spatial position P can be re-expressed as:

[0132]

[0133] By combining the above formulas (23)-(26), the cost function can be expressed as:

[0134]

[0135] in

[0136]

[0137] B=-2w N J 1,M (29)

[0138] C=w N (30)

[0139] As discussed above, it is desirable to obtain the minimum value of the cost function, which can be determined by:

[0140]

[0141] Given:

[0142]

[0143] Finally, the vector Can be determined as:

[0144]

[0145] By calculating the above formula, the gain of the oth audio object among the M clusters can be determined.

[0146] The oth audio object can convert the vector Clustering into M clusters. It can be appreciated that, depending on the determined gain vector, the audio object can be clustered into a cluster in a category into which it is classified, or clustered into a cluster of a different category, or the audio object can be clustered into multiple clusters in a category into which it is classified, or clustered into multiple clusters of multiple different categories.

[0147] When the gain vector When is determined, the reconstructed spatial position of the audio object can be obtained by formula (17). In this regard, the process of determining the gain can also be applied in the cluster assignment based on minimizing the overall spatial distortion as described above to determine the reconstructed spatial position and thus determine the spatial distortion of each category.

[0148] It should be noted that a second order polynomial is used as an example for determining the minimum value of the cost function. In other example embodiments, many other exponent values, such as 1, 1.5, 3, etc., may also be used.

[0149] The fuzzy category clustering method for audio object clustering is described above. In another method, based on reducing the spatial distortion cost associated with each category, the audio objects in the category can be assigned to at least one cluster assigned to the category. That is, cross-category leakage is not allowed. Audio object clustering is performed within each category and audio objects cannot be grouped into clusters assigned to another category. This method can be called a hard category clustering method. In some embodiments in which this method is applied, audio objects can be assigned to more than one cluster in the clusters assigned to the category corresponding to the audio object. In a further embodiment, cross-cluster leakage is not allowed when audio objects are clustered, and an audio object is only assigned to one cluster assigned to the corresponding category.

[0150] The hard class clustering method may be suitable for some specific applications, such as dialogue replacement or dialogue enhancement, which require audio objects (dialogue objects) to be separated from each other.

[0151] In the hard category clustering method, since the audio objects in one category cannot be clustered into one or more clusters of other categories, it is expected that each category is assigned at least one cluster in the previous cluster assignment. Therefore, in some embodiments, the cluster assignment described above by minimizing the overall embarrassment distortion may be more suitable. In other embodiments, when applying category clustering, importance-based cluster assignment can also be used. As described above, some additional conditions can be used in the cluster assignment to ensure that each category is assigned at least one cluster. For example, a minimum threshold of clusters or a minimum threshold of spatial distortion for each category can be used.

[0152] In one or more example embodiments, within a category, an audio object may be assigned to only one cluster or to multiple clusters because the category represents the same type of information to be retained. Figure 2 As shown, audio objects in category 1 can be clustered into one or more of clusters 4, 5, and 6. In a scenario where audio objects are clustered into multiple clusters within a category, corresponding gains can be determined to reduce or even minimize the distortion costs associated with the category, which can be similar to those described for fuzzy category clustering methods. The difference is that the determination is performed within a category. In some embodiments, each input audio object can be allowed to be clustered into only one cluster assigned to its category.

[0153] Two methods for audio clustering are described above. It should be noted that these two methods can be used separately or in combination. For example, after the audio object classification of S101 and the cluster assignment of S102, for some of these categories, the fuzzy category clustering method can be used to cluster their audio objects, and for the remaining categories, the hard category clustering method can be applied. That is, some leakage across categories is allowed within some categories, while leakage across categories is not allowed for other categories.

[0154] After the input audio objects are assigned to the clusters, for each cluster, the audio objects can be combined to obtain clustered audio objects, and the metadata of the audio objects of each cluster can be combined to obtain the metadata of the clustered audio objects. The clustered audio objects can be a weighted sum of all audio objects in the cluster using corresponding gains. The metadata of the clustered audio objects can be the corresponding metadata represented by the category in some examples, or can be the corresponding metadata of any audio object within the cluster or its category or the corresponding metadata of the most important audio object in another example.

[0155] Since all input audio objects are classified into corresponding categories based on the information to be retained in their metadata before audio object clustering, different metadata to be retained or specific combinations of metadata to be retained are associated with different categories. After clustering, for an audio object within a category, it is less likely to be mixed with an audio object associated with a different metadata. In this regard, the metadata of the audio object is retained after clustering. In addition, spatial distortion or distortion cost is taken into account during cluster assignment and audio object allocation.

[0156] Figure 3 The architecture of a system 300 for metadata-preserving audio object clustering according to an example embodiment of the present invention is depicted. Figure 3 As depicted, the system 300 includes an audio object classification unit 301 configured to classify a plurality of audio objects into a first number of a plurality of categories based on information to be retained in metadata associated with the plurality of audio objects. The system 300 also includes a cluster assignment unit 302 configured to assign a predetermined number of clusters to the categories, and an audio object allocation unit 303 configured to assign the audio objects in each of the categories to at least one of the clusters according to the assignment.

[0157] In some embodiments, the information may include at least one or more of: size information of the audio object, region masking information, grab information, content type, and rendering mode.

[0158] In some embodiments, the audio object classification unit 301 may be further configured to classify audio objects without information to be retained into one category, and classify audio objects with different information to be retained into different categories.

[0159] In some embodiments, the cluster assignment unit 302 may further include: an importance-based determination unit configured to determine a predetermined number of audio objects from a plurality of audio objects based on the importance of each audio object relative to other audio objects; and a distribution determination unit configured to determine the distribution of the predetermined number of audio objects between categories. In these embodiments, the cluster assignment unit 302 may further be configured to assign the predetermined number of clusters to the categories based on the distribution.

[0160] In some embodiments, the cluster assignment unit 302 may be further configured to assign a predetermined number of clusters to a class based on reducing the overall spatial distortion of the class.

[0161] In some embodiments, the overall spatial distortion of a category may include the maximum spatial distortion among the individual spatial distortions of the category, or the weighted sum of the individual spatial distortions of the category, and the spatial distortion of each category may be associated with the original spatial position of each audio object in the category and the spatial position of at least one cluster in the cluster.

[0162] In some embodiments, a reconstructed spatial position of each audio object may be determined based on the spatial position of at least one cluster, and a spatial distortion of a category may be determined based on a distance between the original spatial position of each audio object in each category and the reconstructed spatial position of the audio object.

[0163] In some embodiments, multiple audio objects are in a frame of the audio signal, and the spatial distortion of each category may be further based on a difference between the number of clusters assigned to the category in a current frame and the number of clusters assigned to the category in a previous frame.

[0164] In some embodiments, the cluster assignment unit 302 may be further configured to iteratively reduce the overall spatial distortion of a class based on at least one of: an amount of spatial distortion of a class in a previous iteration, or a difference between the spatial distortion of a class in a current iteration and the spatial distortion of the class in a previous iteration.

[0165] In some embodiments, the cluster assignment unit 302 can be further configured to assign a predetermined number of clusters to categories based on at least one of: a first threshold for the number of clusters to be assigned to each category, a second threshold for the spatial distortion with each category, and the importance of each category relative to other categories.

[0166] In some embodiments, the system 300 may further include an audio object reclassification unit configured to reclassify at least one audio object in a category into another category based on the spatial distortion of the category.

[0167] In some embodiments, the audio object allocation unit 303 may be further configured to allocate the audio objects in each category to at least one of the clusters assigned to the category based on reducing the distortion cost associated with each category.

[0168] In some embodiments, the audio object allocation unit 303 may be further configured to allocate the audio objects in each category to at least one of the clusters assigned to one or more of the categories based on reducing the distortion cost associated with the category.

[0169] In some embodiments, the distortion cost is associated with one or more of: an original spatial position of each audio object, a spatial position of at least one cluster, an identification of a category into which each audio object is classified, and an identification of each category to which at least one cluster is assigned.

[0170] In some embodiments, the distortion cost is determined based on one or more of: a distance between an original spatial position of each audio object and a spatial position of at least one cluster, a distance between an original spatial position of each audio object and a reconstructed spatial position of the audio object, the reconstructed spatial position being determined based on the spatial position of at least one cluster, and a mismatch between an identification of a category into which each audio object is classified and an identification of each category to which at least one cluster is assigned.

[0171] In some embodiments, the system 300 may further include: an audio object combination unit configured to combine the audio objects in each cluster to obtain a clustered audio object, and a metadata combination unit configured to combine the metadata of the audio objects in each cluster to obtain the metadata of the clustered audio object.

[0172] For clarity, Figure 3 Certain additional components of system 300 are not depicted. However, it should be understood that the above reference Figure 1The various features described are also applicable to the system 300. Moreover, the various components in the system 300 may be hardware modules, software unit modules, etc. For example, in some embodiments, the system 300 may be partially or entirely implemented using software and / or firmware, such as being implemented as a computer program product contained on a computer-readable medium. Alternatively or additionally, the system 300 may be partially or entirely implemented based on hardware, such as being implemented as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on a chip (SOC), a field programmable gate array (FPGA), etc. The scope of the present invention is not limited in this respect.

[0173] Reference below Figure 4 , which shows a schematic block diagram of a computer system 400 suitable for implementing an embodiment of the present invention. Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. As required, in the RAM 403, data for the CPU 401 to perform various processes, etc. are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0174] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0175] In particular, according to an embodiment of the present invention, the above reference Figure 1 The described process may be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing method 100. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 409, and / or installed from removable media 411.

[0176] In general, various example embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of embodiments of the present invention are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0177] Moreover, each box in the flowchart can be viewed as a method step, and / or an operation generated by the operation of a computer program code, and / or understood as a plurality of coupled logic circuit elements that perform related functions. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly implemented on a machine-readable medium, and the computer program includes a program code configured to implement the method described above.

[0178] Within the disclosed context, a machine-readable medium may be any tangible medium that contains or stores a program for or related to an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of machine-readable storage media include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0179] The computer program code for realizing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when being executed by a computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or the block diagram to be implemented. The program code can be executed completely on a computer, partly on a computer, as an independent software package, partly on a computer and partly on a remote computer or completely on a remote computer or server. The program code can be distributed in a device that is specifically programmed, and these devices can be referred to as "modules" in this article usually. The software grouping parts of these modules can be written in any specific computer language and can be a part of a monolithic integrated code base, or can be developed into multiple discrete code parts, such as usually developed in an object-oriented computer language. In addition, modules can be distributed across multiple computer platforms, servers, terminals, mobile devices, etc. A given module can even be implemented as a single processor and / or computer hardware platform to perform the described function.

[0180] As used in this application, the term "circuitry" refers to all of the following: (a) hardware circuit implementations only (such as analog circuit implementations only and / or digital circuit implementations only) and (b) combinations with circuits and software (and / or firmware), such as (available): (i) a combination with a processor or (ii) a processor / software (including a digital signal processor), software, and a portion of memory that work together to enable a device (such as a mobile phone or server) to perform various functions, and (c) circuits, such as a microprocessor or a portion of a microprocessor, that require software or firmware for operation even if the software or firmware is not physically present. In addition, it is known to those skilled in the art that communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modular data signal, such as a carrier wave or other transport mechanism, and that communication media includes any information transmission media.

[0181] In addition, although the operations are depicted in a particular order, this should not be understood as requiring such operations to be completed in the particular order shown or in a sequential order, or to perform all illustrated operations to obtain the desired result. In some cases, multitasking or parallel processing can be beneficial. Similarly, although the above discussion contains certain specific implementation details, this should not be interpreted as limiting the scope of any invention or claim, but should be interpreted as a description of a specific embodiment that can be directed to a specific invention. Certain features described in the context of separate embodiments in this specification may also be integrated and implemented in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination.

[0182] Various modifications and changes to the aforementioned exemplary embodiments of the present invention will become apparent to those skilled in the relevant art when viewing the aforementioned description in conjunction with the accompanying drawings. Any and all modifications will still fall within the non-limiting and exemplary embodiment scope of the present invention. In addition, the aforementioned description and drawings have the benefit of enlightenment, and those skilled in the art of the art relating to these embodiments of the present invention will think of other embodiments of the present invention set forth herein.

[0183] Thus, the present invention may be implemented in any form described herein.For example, the following Enumerated Example Embodiments (EEE) describe certain structures, features, and functions of certain aspects of the present invention.

[0184] EEE 1. A method for retaining object metadata in audio object clustering, comprising: assigning audio objects to categories, each category representing a specific combination of one or more metadata that need to be retained, and generating multiple clusters for each category through a clustering process, the clustering process complies with the overall (maximum) number of available clusters and the overall error criterion, and the method further includes: fuzzy object category separation or hard object category separation.

[0185] EEE 2. A method according to EEE 1, wherein the fuzzy object category separation method comprises: determining output cluster centers, for example, by selecting the most important objects; and outputting cluster signals by minimizing a cost function that jointly considers: (1) location metadata of each object; (2) The category identifier n of each object o , (3) Location metadata for each cluster and (4) the category identifier n associated with each cluster m .

[0186] EEE 3. A method according to EEE 2, wherein the cost function takes into account the object class identity n o and cluster category identifier n m The mismatch between .

[0187] EEE 4. The method according to EEE 1, wherein the hard object category separation comprises: determining an optimal number of clusters for each category by minimizing overall spatial distortion; and clustering objects within each cluster, the clustering process being performed separately for each category. EEE 4. The method according to EEE 1, wherein the hard object category separation comprises: determining an optimal number of clusters for each category by minimizing overall spatial distortion; and clustering objects within each cluster.

[0188] EEE 5. The method according to EEE 4, wherein the overall spatial distortion includes: spatial positions measuring the difference between original object positions and clustered positions in each category, importance of each category, and changes in the number of clusters for each category.

[0189] EEE 6. According to the method of EEE 4, the process of determining the optimal number of clusters for each category is an iterative process, and in each iteration a cluster is added or assigned to the category that needs it most.

[0190] EEE 7. According to the method of EEE 4, the process of determining the optimal number of clusters for each category further includes object reallocation to avoid large spatial distortion within one category.

[0191] It will be understood that the embodiments of the present invention are not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are used herein, they are used in a generic and descriptive sense only and not for limiting purposes.

Claims

1. A method for decoding an encoded audio signal, comprising: receiving the encoded audio signal and determining an audio object from the encoded audio signal; classifying the audio object into at least one category based on rendering mode information, wherein the rendering mode information is part of metadata associated with the audio object; calculating a spatial distortion metric for the at least one category based on the rendering mode information; assigning a plurality of clusters to the at least one class based on the spatial distortion metric; determining at least one cluster for the at least one category based on the metadata and the assigned plurality of clusters; as well as The audio object is rendered based on the rendering mode information for the at least one cluster.

2. A system for decoding an encoded audio signal, comprising: a receiver for receiving the encoded audio signal and determining an audio object from the encoded audio signal; an audio object classification unit for classifying the audio object into at least one category based on rendering mode information, wherein the rendering mode information is part of metadata associated with the audio object; a spatial distortion metric calculation unit, configured to calculate a spatial distortion metric for the at least one category based on the rendering mode information; a cluster assigning unit, configured to assign a plurality of clusters to the at least one category based on the spatial distortion metric; and determining at least one cluster for the at least one category based on the metadata and the assigned plurality of clusters; as well as A renderer is configured to render the audio object based on the rendering mode information for the at least one cluster.

3. A computer program product, configured to cause a computer to execute the method according to claim 1.

Images