Method and apparatus for clustering audio objects, electronic system, medium and computer program product

Abstract

A method for clustering audio objects can involve identifying a plurality of audio objects, where each of the plurality of audio objects is associated with respective metadata indicative of respective spatial position information and respective rendering metadata. The method can involve assigning audio objects of the plurality of audio objects to a rendering metadata category of a plurality of rendering metadata categories, where at least one rendering metadata category comprises a plurality of rendering metadata types to be maintained. The method can involve determining an allocation of a plurality of audio object clusters to each rendering metadata category. The method can involve rendering audio objects of the plurality of audio objects to the allocated plurality of audio object clusters based on the metadata indicative of spatial position information and based on the assignment of audio objects to rendering metadata categories.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to the following prior applications: International Patent Application PCT / CN2021 / 077110, filed February 20, 2021; U.S. Provisional Patent Application 63 / 165,220, filed March 24, 2021; U.S. Provisional Patent Application 63 / 202,227, filed June 2, 2021; and European Patent Application 21178179.4, filed June 8, 2021, all of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems, methods, and media for clustering audio objects. Background Technology

[0004] Audio content presentation devices capable of displaying spatially positioned audio content are becoming increasingly popular. For example, such devices can present audio content perceived as being located at various spatial positions within the listener's three-dimensional environment. While some existing audio content presentation methods and devices offer acceptable performance under certain conditions, improvements are desirable.

[0005] Symbols and terms Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-emitting transducer (or a group of transducers). A typical set of headphones includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, one or more speaker feeds can undergo different processing in different circuit branches coupled to different transducers.

[0006] Throughout this disclosure, including in the claims, the expression “to” perform an operation on a signal or data (e.g., to filter, scale, transform, or apply gain to a signal or data) is used in a broad sense to indicate performing an operation directly on a signal or data or on a processed version of a signal or data (e.g., a version of a signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0007] Throughout this disclosure, including in the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs, and the other XM inputs are received from an external source) may also be referred to as a decoder system.

[0008] Throughout this disclosure, including in the claims, the term "processor" is used broadly to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) for performing operations on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0009] Throughout this disclosure, including in the claims, the terms "a cluster" or "multiple clusters" are used to refer to a cluster of audio objects. The terms "cluster" and "audio object cluster" should be understood as synonyms and used interchangeably. An audio object cluster is a combination of audio objects having one or more similar attributes (such as audio objects having similar spatial locations and / or similar rendering metadata). In some cases, audio objects may be assigned to a single cluster, while in other cases, audio objects may be assigned to multiple clusters. Summary of the Invention

[0010] At least some aspects of this disclosure can be implemented via methods. Some methods may involve identifying a plurality of audio objects, wherein each of the plurality of audio objects is associated with corresponding metadata indicating corresponding spatial location information and corresponding rendering metadata. Some methods may involve assigning audio objects from the plurality of audio objects to rendering metadata categories from a plurality of rendering metadata categories, wherein at least one rendering metadata category includes a plurality of rendering metadata types to be maintained. Some methods may involve determining an allocation of a plurality of audio object clusters to each rendering metadata category, wherein an audio object cluster includes one or more audio objects from the plurality of audio objects having similar properties. Some methods may involve rendering audio objects from the plurality of audio objects to the allocated plurality of audio object clusters based on metadata indicating spatial location information and based on the assignment of audio objects to rendering metadata categories.

[0011] In some examples, the rendering metadata categories include a bypass mode category and a virtualization category. In some examples, multiple rendering metadata types included in the virtualization category include multiple virtualization types, each representing the distance from the head center to the audio object.

[0012] In some examples, the rendering metadata category includes either the region category or the capture category.

[0013] In some examples, audio objects assigned to the first rendering metadata category are prohibited from being assigned to audio object clusters in the multiple audio object clusters that are assigned to the second rendering metadata category.

[0014] In some examples, determining the allocation of the plurality of audio object clusters to each rendering metadata category involves: (i) determining an initial allocation of the initial plurality of audio object clusters to each rendering metadata category; (ii) assigning audio objects to the initial plurality of audio object clusters based on metadata indicating spatial location information and based on the assignment of audio objects to rendering metadata categories; (iii) determining the category cost for assigning audio objects to the initial plurality of audio object clusters for each rendering metadata category; (iv) determining an updated allocation of the initial plurality of audio object clusters to each rendering metadata category based at least in part on the category cost of each rendering metadata category; and (iv) repeating (ii) to (iv) until a stopping criterion is met. In some examples, determining the category cost for assigning audio objects to the initial plurality of audio object clusters is based on the position of the audio object clusters assigned to the rendering metadata category and the position of the audio objects assigned to the audio object clusters assigned to that rendering metadata category. In some examples, the category cost is based on the left-right placement of the audio object relative to the left-right placement of the audio object cluster to which the audio object has been assigned. In some examples, determining the category cost for assigning audio objects to the initial plurality of audio object clusters is based on the loudness of the audio object. In some examples, determining the category cost of assigning an audio object to an initial plurality of audio object clusters is based on the distance of the audio object to the audio object cluster to which it has already been assigned. In some examples, determining the category cost of assigning an audio object to an initial plurality of audio object clusters is based on the similarity between the rendering metadata type of the audio object and the rendering metadata type of the audio object cluster to which it has already been assigned. In some examples, the method may involve determining a global cost based on the category cost of each rendering metadata category, wherein the update allocation of the initial plurality of audio object clusters is based on the global cost. In some examples, determining the update allocation includes changing the number of audio object clusters assigned to at least one of the plurality of rendering metadata categories. In some examples, the method may further involve determining a global cost based on the category cost of each rendering metadata category, wherein the number of audio object clusters is determined based on the global cost. In some examples, determining the number of audio object clusters includes minimizing the global cost under a constraint on the number of audio object clusters, indicating the maximum number of audio object clusters that can be added.

[0015] In some examples, rendering an audio object from a plurality of audio objects to an assigned plurality of audio object clusters includes determining the object-to-cluster gain of each of the plurality of audio objects when rendered to one or more audio object clusters assigned to the rendering metadata class to which the audio object is assigned. In some examples, the object-to-cluster gain of an audio object assigned to a first category of the plurality of rendering metadata classes is determined separately from the object-to-cluster gain of an audio object assigned to a second category of the plurality of rendering metadata classes. In some examples, the object-to-cluster gain of an audio object assigned to a first category of the plurality of rendering metadata classes and the object-to-cluster gain of an audio object assigned to a second category of the plurality of rendering metadata classes are determined jointly.

[0016] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more non-transitory media on which software is stored.

[0017] At least some aspects of this disclosure can be implemented via an apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0018] This disclosure provides various technical advantages. For example, audio objects that can be associated with spatial location information and rendering metadata indicating how the audio objects will be rendered can be clustered in a way that preserves rendering metadata across different rendering metadata categories. In some cases, when clustering audio objects within the same rendering metadata category, rendering metadata may not be preserved. By using a hybrid approach of preserving rendering metadata based on rendering metadata categories to cluster audio objects, the techniques described herein allow the generation of audio signals with clustered audio objects, which reduces spatial distortion during the rendering of the audio signal and reduces the bandwidth required to transmit such audio signals. Such audio signals can advantageously be more faithful to the intent of the audio content creator associated with the audio signal.

[0019] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the specification, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0020] Figure 1A and Figure 1B The illustration shows a representation of an example audio object cluster based on rendering metadata and spatial location metadata according to some implementation methods.

[0021] Figure 2 An example of a process for clustering audio objects based on spatial location metadata while preserving rendering metadata is shown, according to some implementations.

[0022] Figure 3 An example of a process for determining cluster allocation according to some implementations is shown.

[0023] Figure 4 An example of a process for assigning audio objects to an assigned cluster, according to some implementations, is shown.

[0024] Figure 5 A block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0025] In the various figures, similar reference numerals and names indicate similar elements. Detailed Implementation

[0026] Audio content presentation devices capable of displaying spatially positioned audio content (whether via loudspeakers or headphones) are becoming increasingly popular. For example, such devices can present audio content perceived as being located at various spatial positions within a listener's three-dimensional environment. This audio content can be encoded in an audio format that includes an "audio bed" and "audio objects." The audio bed comprises the audio content to be rendered at a fixed spatial position, while the audio objects comprise the audio content that can be rendered at varying spatial positions and / or for different durations. For example, an audio object could represent sound effects associated with a moving object (e.g., a buzzing insect, a moving vehicle, etc.), music from a moving instrument (e.g., moving instruments in a marching band, etc.), or other audio content that may move in location.

[0027] Each audio object can be associated with metadata describing how the audio object will be rendered (generally referred to herein as "rendering metadata") and / or the spatial location that the audio object will be perceived to be in when rendered (generally referred to herein as "spatial location metadata"). For example, spatial location metadata can indicate the location of the audio object in three-dimensional (3D) space that will be perceived by the listener when rendered. Spatial location metadata can specify the azimuth position and / or elevation position of the audio object. As another example, rendering metadata can indicate how the audio object will be rendered. It should be noted that the example rendering metadata type used for headphone rendering mode can be different from the rendering metadata type used for speaker rendering mode. In some implementations, rendering metadata can be associated with rendering metadata categories. For example, the rendering metadata associated with headphone rendering mode can be associated with a first category corresponding to the "bypass mode" category and a second category corresponding to the "room virtualization" category, in which room virtualization is not applied when rendering is assigned to audio objects of the first category, while in the "room virtualization" category, room virtualization techniques are applied when rendering is assigned to audio objects of the second category. Continuing this example further, in some embodiments, a rendering metadata category may have multiple rendering metadata types within that category. As a more specific example, the rendering metadata associated with the “Room Virtualization” category of rendering metadata may have multiple rendering metadata types, such as “Near,” “Medium,” and “Far,” each indicating the relative distance from the listener’s head to the location within the room where the audio object is to be rendered. As another example, the rendering metadata associated with a speaker rendering mode may be associated with a first rendering metadata category corresponding to a “snap” mode and a second rendering metadata category corresponding to a “zone mask” mode, where the snap mode indicates that the audio object is rendered to a specific speaker for point-source rendering, and the zone mask mode indicates that the audio object is not rendered to a specific speaker included in a specific speaker group (generally referred to herein as a “zone mask”). As a more specific example, in some embodiments, the “snap” category of rendering metadata may include a rendering metadata type corresponding to a specific speaker. In some embodiments, the “snap” category of rendering metadata may include a binary value, wherein, in response to the rendering metadata being “1” or “Yes” (indicating that “snap” is enabled), the audio object may be rendered by the nearest speaker. As another more specific example, the “region masking” category of rendering metadata can include rendering metadata types that correspond to different speaker groups that will not be used to render audio objects (e.g., “left surround and right surround”, “left and right”, etc.).In some embodiments, the “region masking” category of the rendering metadata can indicate one or more speakers to which an audio object should be rendered (e.g., “front”, “rear”, etc.), and other speakers will be excluded or prevented from rendering the audio object.

[0028] The metadata associated with audio objects (whether spatial location metadata or rendering metadata) can be specified by the audio content creator and thus represent the creator's artistic vision. Therefore, maintaining spatial location metadata and / or rendering metadata to faithfully represent the creator's artistic vision can be crucial. However, in some cases, such as the soundtracks of movies or television programs, audio content can include dozens or hundreds of audio objects. Therefore, audio content formatted to include audio objects can be very large and quite complex. Transmitting such audio content for rendering can be difficult and may require significant bandwidth. This increased bandwidth requirement can be particularly problematic for viewers or listeners at home, who may be more constrained by bandwidth considerations when watching or listening to such audio content at home compared to cinemas.

[0029] To reduce the complexity of audio content, audio objects can be clustered, at least in part, based on spatial positioning metadata, so that audio objects with relatively close locations (e.g., azimuth and / or elevation) are assigned to the same audio object cluster. The audio object clusters can then be transmitted and / or rendered. By using aggregated metadata associated with the audio object clusters to render audio objects assigned to the same cluster, spatial complexity can be reduced, thereby decreasing the bandwidth used for transmitting and / or rendering audio signals.

[0030] However, clustering audio objects without considering rendering metadata and the category to which each audio object has been assigned can introduce perceptual discontinuities. For example, assigning an audio object to the “bypass mode” category of rendering metadata to a cluster associated with the “room virtualization” category of rendering metadata can lead to perceptual distortion, even if that audio object and other audio objects assigned to that cluster are associated with similar azimuth and / or elevation spatial locations. In particular, by assigning an audio object to a cluster associated with the “room virtualization” category of rendering metadata, the audio object can undergo a transformation using the Head Relative Transfer Function (HRTF) to simulate the propagation path from the source to the listener's ear. The HRTF transformation can distort the perceptual quality of the audio object, for example, by introducing timbre variations associated with the rendered audio object and / or by introducing temporal discontinuities when several frames of audio content are assigned to different categories. Furthermore, because the first audio object is assigned to the "bypass mode" category by the audio content creator, rendering the first audio object using the HRTF that is to be applied to the audio object assigned to the "room virtualization" category may cause the first audio object to be rendered in a way that is not faithful to the intent of the audio content creator.

[0031] Clustering audio objects in a way that strictly preserves rendering metadata categories and / or rendering metadata types within specific rendering metadata categories can also have certain consequences. For example, clustering audio objects using strictly preserved rendering metadata may require a relatively large number of clusters, which increases the complexity of the audio signal and may require higher bandwidth for audio signal encoding and transmission. Alternatively, clustering audio objects using strictly preserved rendering metadata with a limited number of clusters may result in spatial distortion because two audio objects with the same rendering metadata but located relatively far apart from each other may be rendered into the same cluster.

[0032] The techniques, systems, methods, and media described herein describe the assignment and / or generation of audio object clusters that, in some cases, retain rendering metadata categories, while in others, allow clustering of audio objects associated with a specific rendering metadata category or a specific rendering metadata type within a rendering metadata category, and audio objects associated with different rendering metadata categories or different rendering metadata types. The techniques, systems, methods, and media described herein can allow for reduced spatial complexity by clustering audio objects, thereby reducing the bandwidth required to transmit and / or render such audio objects, while also improving the perceptual quality of rendered audio objects by retaining rendering metadata in some cases and not in others. Specifically, by allowing flexible use of rendering metadata categories or types when assigning audio objects to audio object clusters, spatial distortion caused by strict rendering metadata constraints during clustering can be reduced or eliminated, while still achieving a reduction in audio content complexity, thereby reducing the bandwidth required to transmit such audio content. Audio object clusters can be considered to be associated with audio objects having similar attributes, where similar attributes may include similar spatial locations and / or similar rendering metadata (e.g., the same rendering metadata category, the same rendering metadata type, etc.). Spatial similarity can be determined based on the distance between an audio object and the centroid of the cluster to which it is assigned (e.g., Euclidean distance and / or any other suitable distance metric). In embodiments where an audio object can be rendered into multiple audio object clusters, the audio object can be associated with multiple weights, each corresponding to an audio object cluster, where the weights indicate the extent to which the audio object is rendered into a particular cluster. Continuing this example, when an audio object is relatively far from a particular audio object cluster (e.g., the spatial location associated with the audio object is relatively far from the centroid associated with the audio object cluster), the weight associated with the audio object cluster can be relatively small (e.g., close to or equal to 0). In some embodiments, two audio objects can be considered to have similar properties based on the similarity of the weights associated with each of the two audio objects, indicating the extent to which each audio object is rendered into a particular audio object cluster.

[0033] In some implementations, audio object clusters can be generated such that audio objects assigned to a specific rendering metadata category (e.g., "bypass mode") are prohibited from being assigned to clusters containing audio objects assigned to other rendering metadata categories (e.g., "virtualization mode"). In some such implementations, audio objects within a specific rendering metadata category can be assigned to clusters containing audio objects with the same rendering metadata type within that specific category and / or clusters containing audio objects with different rendering metadata types within that specific category. For example, in some implementations, a first audio object assigned to the "virtualization mode" category and having a "near" rendering metadata type (e.g., indicating that the first audio object will be rendered relatively close to the listener's head) can be assigned to a cluster including a second audio object assigned to the "virtualization mode" category and having a "medium" rendering metadata type (e.g., indicating that the second audio object will be rendered at a medium distance from the source to the listener's head). Continuing with this example, in some implementations, the first audio object may be prevented from being assigned to a cluster that includes a third audio object, which is assigned to the “virtualization mode” category and has a “far” rendering metadata type (e.g., indicating that the third audio object will be rendered relatively far from the listener’s head).

[0034] Figure 1A Example 100 of a representation of clustering of audio objects is shown, wherein clustering of audio objects assigned to a particular rendering metadata category with audio objects assigned to other rendering metadata categories is not allowed.

[0035] In Example 100, there are two rendering metadata categories. Category 102 (in...) Figure 1A The category 1 (represented as "Category 1") corresponds to the audio object associated with the "Bypass Mode" rendering metadata. Category 104 (in...) Figure 1A The "Category 2" category corresponds to the audio object associated with the "Virtualization Mode" rendering metadata. The "Virtualization Mode" category of rendering metadata can have various potential rendering metadata types, such as "near," "medium," and / or "far" distance from the listener's head. Therefore, an audio object assigned to the "Virtualization Mode" category of rendering metadata can have a rendering metadata type selected from "near," "medium," or "far" (e.g., ...). Figure 1A As shown and as Figure 1A (Depicted by the shadow type applied to each audio object).

[0036] Figure 1AThis shows a group of audio objects (e.g., audio object 106) that have been clustered based on spatial location metadata associated with the audio objects and on rendering metadata categories associated with the audio objects. The assigned clusters are indicated by numbers within circles depicting each audio object. For example, as... Figure 1A As shown, audio object 106 has been assigned to cluster "1". As another example, within category 104, audio object 108 has been assigned to cluster "4".

[0037] exist Figure 1A In Example 100, the rendering metadata category is strictly maintained when generating audio object clusters. For example, audio objects assigned to the “Bypass Mode” category of the rendering metadata are prohibited from being assigned to clusters assigned to the “Virtualization Mode” category of the rendering metadata. Similarly, audio objects assigned to the “Virtualization Mode” category of the rendering metadata are prohibited from being assigned to clusters assigned to the “Bypass Mode” category of the rendering metadata.

[0038] exist Figure 1A In Example 100, audio objects assigned to a specific rendering metadata category can be clustered with other audio objects assigned to the same rendering metadata category but having different rendering metadata types within that category. For example, within category 104, audio object 110 associated with a "near" rendering metadata type within the "virtualization mode" category and audio objects 112 and 114, each associated with a "medium" rendering metadata type within the "virtualization mode" category, can be clustered. As another example, within category 104, audio object 116 associated with a "medium" rendering metadata type within the "virtualization mode" category and audio objects 118 and 120, each associated with a "far" rendering metadata type within the "virtualization mode" category, can be clustered.

[0039] It should be noted that the clustering of audio objects depicted in Example 100 can be the result of a clustering algorithm or technique. For example, the clustering of audio objects depicted in Example 100 can be used in… Figure 2 The process 200 is illustrated in the diagram and described below in conjunction with the techniques used to generate the process. In some implementations, optimization algorithms or techniques may be used to determine the allocation to... Figure 1A The number of audio object clusters for each category and / or the spatial centroid location of each cluster are shown. For example, this can be achieved using... Figure 3 The technique shown in process 300 and described below in conjunction with this process iteratively determines the allocation of audio object clusters to generate an optimal allocation. Additionally, in some implementations, assigning audio objects to specific clusters can be accomplished by determining the object-to-cluster gain, which describes the ratio or gain of an audio object when rendered to a specific cluster, as described below in conjunction with... Figure 4 The process described in 400.

[0040] In comparison, Figure 1B Example 150 shows a representation of clustering of audio objects, where, in some cases, audio objects that would be assigned to a particular rendering metadata category are allowed to be assigned to clusters that would be assigned to other rendering metadata categories.

[0041] like Figure 1B As illustrated, audio objects assigned to a specific rendering metadata category can be assigned to clusters allocated to different rendering metadata categories. For example, audio objects 152 and 154, each assigned to the "virtualization mode" category, are assigned to the "bypass mode" category (e.g., ...). Figure 1B The cluster of category 102). As another example, audio objects 156 and 158, each assigned to the “bypass mode” category, are assigned to the “virtualization mode” category (e.g., Figure 1B The cluster of category 104.

[0042] It should be noted that, although Figure 1A and Figure 1B This shows that each audio object is assigned to a single cluster, but audio objects can be assigned to or rendered to multiple clusters (as combined below). Figure 2 and Figure 4 (As described). The degree to which a particular audio object is assigned and / or rendered to a particular cluster is generally referred to in this paper as "object-to-cluster gain". For example, for an audio object... j and cluster c An object-to-cluster gain of 1 indicates an audio object. j Fully assigned or rendered to a cluster c As another example, an object-to-cluster gain of 0.5 represents an audio object. j Assigned or rendered to a cluster with a gain of 0.5 c and with audio objects j The associated residual signal is rendered to other clusters. As yet another example, an object-to-cluster gain of 0 indicates an audio object. j Not assigned or rendered to cluster c .

[0043] Figure 2An example of process 200, according to some embodiments, for assigning clusters to different rendering metadata categories and assigning audio objects to the assigned clusters. Process 200 can be executed on various devices, such as on a server that encodes an audio signal based on audio objects and associated metadata provided by the audio content creator. It should be noted that process 200 describes the process in general terms for a single frame of audio content. However, it should be understood that in some embodiments, the blocks of process 200 may be repeated for one or more other frames of the audio content, for example, to generate a compressed version of the complete output audio signal as the input audio signal. In some implementations, one or more blocks of process 200 may be omitted. Additionally, in some implementations, two or more blocks of process 200 may be executed substantially in parallel. The blocks of process 200 may be executed in any order, and are not limited to this. Figure 2 The order shown.

[0044] Process 200 can begin at 202, which involves identifying a set of audio objects, each associated with spatial location metadata and rendering metadata. Audio objects in this set can be identified for a specific frame of the input audio signal. Audio objects can be identified, for example, by accessing a list or table associated with a frame of the input audio signal. Spatial location metadata can indicate spatial location information (e.g., position in 3D space) associated with rendering the audio object. For example, spatial location information can indicate the azimuth and / or elevation position of the audio object. As another example, spatial location information can indicate Cartesian coordinates (e.g.,...). x , y , z The spatial location of an audio object in coordinates. Rendering metadata can indicate how an audio object will be rendered.

[0045] At position 204, procedure 200 can assign each audio object to a rendering metadata category. Example rendering metadata categories for headphone rendering mode include the "Bypass Mode" and "Virtualization Mode" categories. Example rendering metadata categories for speaker rendering mode include the "Capture Mode" and "Region Masking" categories. Within a rendering metadata category, rendering metadata can be associated with a rendering metadata type.

[0046] In some implementations, at least one rendering metadata category may include one or more (e.g., two, three, five, ten, etc.) rendering metadata types. In headphone rendering mode, example rendering metadata types within the “Virtualization Mode” category of rendering metadata include “Near,” “Middle,” and “Far” virtualization. It should be noted that the rendering metadata types within the “Virtualization Mode” category of rendering metadata may indicate a specific HRTF to be applied to the audio object to produce the virtualization indicated in the rendering metadata. For example, the rendering metadata corresponding to “Near” virtualization may specify the use of a first HRTF, while the rendering metadata corresponding to “Middle” virtualization may specify the use of a second HRTF. Example rendering metadata types within the “Capture” category of rendering metadata may include binary values ​​indicating whether capture is enabled and / or specific identifiers for the speakers to which the audio object is to be rendered (e.g., “Left Speaker,” “Right Speaker,” or any other specific speaker). Example rendering metadata types within the “Region Masking” category of rendering metadata include “Left Surround and Right Surround,” “Left Speaker and Right Speaker,” or any other suitable combination of speakers indicating whether they are to be included in or excluded from the rendering of the audio object.

[0047] At 206, process 200 can determine the allocation of clusters to each rendering metadata category. Process 200 can determine the allocation of clusters to each rendering metadata category such that a certain number of clusters allocated to each category optimally contain the audio objects in the set of audio objects identified at box 202 and are subject to any suitable constraints. For example, process 200 can determine the cluster allocation such that the total number of clusters across all rendering metadata categories is less than or equal to a predetermined maximum number of clusters (generally denoted herein as ). M total In some embodiments, the predetermined maximum number of clusters under all rendering metadata categories can be determined based on various criteria or requirements, such as the bandwidth required to transmit encoded audio signals with the predetermined maximum number of clusters.

[0048] As another example, process 200 can determine cluster assignment by iteratively optimizing the cluster assignment based at least in part on a cost function associated with the audio object to be assigned to each cluster. In some embodiments, the cost function can represent various criteria, such as the distance of the audio object assigned to a particular cluster to the centroid of that cluster, the loudness of the audio object relative to the expected loudness of that audio object when rendered to that particular cluster (e.g., as indicated by the audio content creator), etc. Various criteria that can be incorporated into the cost function are discussed below. Figure 3A more detailed description follows. In some implementations, clusters can be assigned based on the assumption that audio objects to be assigned to a particular category are not allowed to be assigned to clusters assigned to different categories. It should be noted that an example of the process used to determine the assignment of audio object clusters to each render metadata category is provided in [the document / document / etc.]. Figure 3 The following is shown and combined Figure 3 Describe it.

[0049] At 208, process 200 may assign and / or render audio objects to assigned clusters based on spatial location metadata and the assignment of audio objects to rendering metadata categories. Assigning and / or rendering audio objects to assigned clusters based on spatial location metadata may involve assigning audio objects to clusters based on the spatial location of the audio object relative to the spatial location of the assigned cluster (e.g., elevation and / or azimuth, Cartesian coordinates, etc.). For example, in some embodiments, process 200 may assign and / or render audio objects to assigned clusters based on spatial location metadata and based on the centroid of each assigned cluster, such that audio objects with similar spatial locations are assigned to the same cluster. In some embodiments, the similarity of the spatial locations of audio objects may be determined based on the distance between the spatial location indicated in the spatial location metadata associated with the audio object and the centroid of the cluster (e.g., Euclidean distance, etc.).

[0050] Assigning and / or rendering audio objects to assigned clusters based on their association with rendering metadata categories can involve maintaining the rendering metadata category by assigning the audio objects to clusters associated with the same rendering metadata category. For example, in some embodiments, process 200 may assign audio objects to assigned clusters such that, as Figure 1A As shown and described above in conjunction with Figure 1, audio objects assigned to a first rendering metadata category (e.g., "bypass mode") are prohibited from being assigned and / or rendered to clusters assigned to a second rendering metadata category (e.g., "virtualization mode"). In some implementations, assigning and / or rendering audio objects to assigned clusters based on their assignment to rendering metadata categories may involve allowing audio objects to be assigned to clusters associated with different rendering metadata categories. For example, in some embodiments, process 200 may assign and / or render audio objects to assigned audio object clusters such that, as Figure 1B Shown and combined above Figure 1BAs described, audio objects assigned to a first rendering metadata category (e.g., "bypass mode") may be assigned to a cluster of audio objects assigned to a second rendering metadata category (e.g., "virtualization mode"). For example, cross-category assignment of audio objects may be desirable where it reduces spatial distortion (e.g., due to the location of the audio object cluster relative to the audio object's position). It should be noted that cross-category assignment of audio objects may introduce timbre variations in the perceived quality of the audio objects when rendered to clusters of audio objects associated with different rendering metadata categories. As another example, in some embodiments, process 200 may assign audio objects such that clustering is allowed for audio objects associated with a first rendering metadata type (e.g., "near" virtualization) within a specific rendering metadata category and other audio objects associated with a second rendering metadata type (e.g., "medium" virtualization), as per the description of... Figure 1A and Figure 1B Category 104 is shown in [reference needed]. It should be noted that the example process for assigning and / or rendering audio objects to an assigned cluster of audio objects subject to various constraints is [reference needed]. Figure 4 Shown in and combined below Figure 4 Describe it.

[0051] Assigning and / or rendering audio objects to a specific cluster can include determining the audio object-to-cluster gain, which indicates the gain applied to an object when it is rendered as part of an audio object cluster. For a given audio object j and audio object clusters c The audio object-to-cluster gain is generally expressed as follows in this paper: As mentioned above, it should be noted that audio objects j It can be rendered to multiple audio object clusters, where, for a specific audio object j and specific clusters c The audio object is converted to a cluster gain indicator when the audio object is... j As a cluster c Gain applied to audio objects during part of the rendering process 。 In some implementations, gain It can be in the range of 0 to 1, where the value indicates when an audio object is... j Rendered to audio object cluster c When to be applied to the audio object j The ratio of the input audio signal. In some implementations, in all clusters c On a specific audio object j The sum of the gains is 1, which indicates that the audio object... j The associated input audio signal must be distributed across all clusters.

[0052] Figure 3 An example of a process 300 for generating cluster assignments across multiple rendering metadata categories, according to some embodiments, is shown. The frames of process 300 can be implemented on any suitable device, such as a server that generates an encoded audio signal based on audio objects included in the input audio signal. It should be noted that process 300 describes the process in general terms for a single frame of audio content; however, it should be understood that in some embodiments, the frames of process 300 can be repeated for one or more other frames of the audio content, for example, to obtain cluster assignments for multiple frames of the audio content. In some embodiments, one or more frames of process 300 can be omitted. Additionally, in some embodiments, two or more frames of process 300 can be executed substantially in parallel. The frames of process 300 can be executed in any order, and are not limited to this. Figure 3 The order shown.

[0053] Generally, process 300 may begin with the initial assignment of clusters to render metadata categories. In some implementations, process 300 may iteratively loop through boxes 304-318 described below to optimally assign clusters to render metadata categories after the initial assignment begins. In some implementations, the assignment may be optimized by minimizing a global cost function that combines the cost functions of each render metadata category. The cost function of a render metadata category is generally referred to herein as the "intra-category cost function". The intra-category cost function of a render metadata category may indicate the cost associated with assigning an audio object to a specific cluster assigned to the render metadata category during the current iteration through boxes 304-318. In some implementations, the intra-category cost function may be based on a corresponding intra-category penalty function, as described below in conjunction with box 314. The intra-category penalty function may depend on one or more intra-category penalty terms, as described below in conjunction with boxes 304-310. Each intra-category penalty term may depend, in turn, on the specific audio object. j and cluster c The audio object to cluster gain, which is generally expressed in this paper as The object-to-cluster gain can be determined by minimizing the total intra-group penalty function for a specific rendering metadata category (e.g., as described below in conjunction with box 312), where the total intra-group penalty function associated with that category is the sum of the individual intra-category penalty terms. In other words, process 300 can determine, via boxes 304-312 of process 300, the object-to-cluster gain that minimizes the intra-category penalty function for each rendering metadata category for the current allocation of clusters to rendering metadata categories during the current iteration via boxes 304-318. The object-to-cluster gain can be used to determine the intra-category cost function for each rendering metadata category. The intra-category cost functions can then be combined to generate a global cost function. Clusters can then be reallocated by minimizing the global cost function.

[0054] Process 300 may begin at 302, which determines the initial allocation of clusters to rendering metadata categories, wherein each rendering metadata category is allocated a subset of clusters. In some implementations, clusters may be allocated such that the total number of allocated clusters is less than or equal to a predetermined maximum number of clusters, generally referred to herein as... M total For example, in the first rendering metadata category, there are assigned... m Each cluster and the second rendering metadata category are assigned n In the case of a cluster, m+n ≤ M total . M total The clustering can be determined based on any suitable criterion, such as the total number of audio objects to be clustered, or the available bandwidth for transmitting coded audio signals based on the clustered audio objects. For example, it can be determined that... M total This makes it have M total The bandwidth used by each cluster to transmit encoded audio signals is less than a threshold bandwidth. In some implementations, each rendering metadata category may be assigned at least one cluster.

[0055] Process 300 can determine the centroid of each initially assigned cluster. For example, in some implementations, the centroid of a cluster can be determined based on the perceptually most salient audio object assigned to the rendering metadata category associated with the cluster. As a more specific example, for initially assigned clusters with... m The first rendering metadata category for each cluster (e.g., "bypass mode") m The centroid of each cluster can be determined, at least in part, based on the perceptual saliency of audio objects assigned to the first rendering metadata category. For example, in some implementations, the centroid of an audio object assigned to the first rendering metadata category can be identified. m The most perceptually salient audio object. This m The most perceptually salient audio object can be identified based on various criteria, such as its loudness, spatial distance from other audio objects assigned to the first rendering metadata category, and timbre differences associated with audio objects in the first rendering metadata category. In some implementations, the perceptual salientity of an audio object can be determined based on differences between audio objects. For example, for audio objects that include speech content, if the speech content associated with two audio objects is in different languages, the two audio objects can be determined to be perceptually salient to each other. The centroid of the cluster of audio objects assigned to each rendering metadata category can be determined in a similar manner.

[0056] At 304, process 300 may generate a first intra-category penalty for each rendering metadata category, which indicates the difference between the location of an audio object assigned or rendered to an initially assigned audio object cluster in that category and the location of the initially assigned audio object cluster (e.g., centroid location).

[0057] audio object j The position is generally referred to in this article as p j In some implementations, the audio object j The location is specified by the audio content creator. Cluster c The position is generally referred to in this article as p c .cluster c The location can indicate the cluster c The location of the centroid is as described in the above combined box 302.

[0058] audio object j The reconstructed location after being rendered to one or more clusters is generally referred to in this paper as... Used for calculation An example of the equation is given by the following formula:

[0059] In some implementations... p j , p c and It can represent an audio object j A three-dimensional vector representing the spatial location when rendered to one or more clusters. This spatial location can be represented in Cartesian coordinates.

[0060] The first category of penalty can indicate the aggregated difference between the location of an audio object when it is assigned or rendered to one or more clusters and the original location of the audio object (generally referred to in this article as...). E p An example equation for determining the first-category penalty term, which indicates the aggregated difference between the location of an audio object when rendered to one or more clusters and its original location, is given by:

[0061] It should be noted that the first category penalty item described above, as well as the other category penalty items described below in conjunction with boxes 306-310, are generally related to a single audio object. j This is described as follows: Intra-category penalties can be calculated for each audio object, and the sum can be calculated across all audio objects assigned to a specific rendering metadata category.

[0062] At 306, process 300 may generate a second intra-category penalty for each render metadata category. This second intra-category penalty indicates the distance between the audio object assigned or rendered to the initially allocated cluster in that category and the cluster in that category. The second intra-category cost is generally referred to herein as... E D Costs within Category Two E D Can be based on audio objects j With audio objects j The cluster assigned c The distance between them is measured to determine this. Used for calculation. E D Example equations are given by the following:

[0063] In the equation above, Represents an audio object j Location and cluster c The distance between positions. Because audio objects positioned in the left region will generate perceptual artifacts when rendered to clusters in the right region (or vice versa), audio objects... j Location and cluster c The distance between positions can effectively penalize audio objects during binaural rendering. j Assigned to clusters located in different hemispheres c The modified distance. Used to calculate audio objects. j with cluster c An example equation for the modified distance between them is given by the following:

[0064] In the equation above, Λ can represent a 3 × 3 diagonal matrix, which is given by the following equation:

[0065] In the above formula, λ xx It can depend on the audio object j and clusters c The position varies depending on whether it is located in different left / right regions. Used to determine λ xx An example of the equation for the value of is given by the following formula:

[0066] In the above formula, x j and x c These represent the x-coordinates of the audio object's position and the cluster's position, respectively. In the above formula,a It is a constant between 0 and 1.

[0067] At 308, process 300 can generate a third-category intra-penalty term for each render metadata category. This third-category intra-penalty term indicates the loudness preservation of an audio object when it is assigned or rendered to the various clusters allocated to the render metadata category. In other words, the third-category intra-penalty term can indicate the change in energy or amplitude of an audio object when it is rendered to the various clusters, where energy or amplitude is perceived as loudness by the listener. Therefore, by minimizing the third-category intra-penalty term, perceptual artifacts introduced by rendering audio objects with enhanced or diminished amplitude (and thus enhanced or diminished loudness) can be minimized. The third-category intra-penalty term is generally referred to herein as... E N An example equation for calculating the penalty term within the third category is given below:

[0068] In some implementations, at 310, process 300 may generate a fourth-category penalty term indicating a mismatch between the rendering metadata type associated with the audio object and the rendering metadata type of the cluster to which the audio object is assigned or rendered. It should be noted that box 310 may be omitted for categories that do not include multiple rendering metadata types within a rendering metadata category. For example, for the “bypass mode” category of rendering metadata, the fourth-category penalty term may not be calculated.

[0069] As an example, in the case of headphone rendering, the fourth-category penalty term can indicate a mismatch between the virtualization type (e.g., "near," "medium," or "far") associated with the "virtualization mode" category of the audio object's rendering metadata and the virtualization type of one or more clusters to which the audio object is assigned or rendered. In practice, the fourth-category penalty term can penalize, for example, assigning an audio object with a specific virtualization type (e.g., "near," "medium," or "far") to a cluster associated with a different virtualization type. In some implementations, the penalty amount can depend on the distance between the different virtualization types. For example, assigning a first audio object with a "near" virtualization type to a cluster associated with a "far" virtualization type could be associated with a larger penalty than assigning a second audio object with a "near" virtualization type to a cluster associated with a "medium" virtualization type. The term used to calculate the fourth-category penalty term (generally referred to herein as...) E G An example of the equation is:

[0070] In the equations given above, U HRM(j)HRM(c) Can represent a matrixU The element that defines the audio object. j and cluster c Penalty weights for various combinations of virtualization types. Matrix U Each row can indicate the virtualization type associated with the audio object, and the matrix U Each column can indicate the virtualization type associated with the cluster to which the audio object has been assigned or rendered. For example, matrix elements [ HRM(j) , HRM(c) [Can indicate by] HRM(j) The virtualized type of audio object indicated j When assigned or rendered to a virtualized type HRM(c) clusters c Penalty weight at time 。 In some implementations, the matrix U It can be symmetric, such that the same penalty weight is applied to audio objects of the first virtualization type when assigned or rendered to a cluster of the second virtualization type, and to audio objects of the second virtualization type when assigned or rendered to a cluster of the first virtualization type. In some implementations, the matrix... U The diagonal can be 0, which indicates the similarity between the virtualization type associated with the audio object and the virtualization type associated with the cluster. Matrices that can be used... U Specific examples are as follows:

[0071] At 312, process 300 can determine the object-to-cluster gain for each audio object and the cluster assigned to the rendering metadata category associated with that audio object. The object-to-cluster gain can be determined by minimizing the category penalty function corresponding to the rendering metadata category associated with the audio object. For example, for an audio object associated with the “Bypass Mode” category of the rendering metadata, the object-to-cluster gain of that audio object can be determined for one or more clusters assigned to the “Bypass Mode” category of the rendering metadata. As another example, for an audio object associated with the “Virtualization Mode” category of the rendering metadata, the object-to-cluster gain of that audio object can be determined for one or more clusters assigned to the “Virtualization Mode” category of the rendering metadata.

[0072] The category penalty function for a specific rendering metadata category can be determined as the sum (e.g., a weighted sum) of penalty terms within any of the categories defined in boxes 304-310. For example, in some implementations, the category penalty function for the “virtualization mode” category of rendering metadata can be a weighted sum of penalty terms within a first category defined in box 304, a second category defined in box 306, a third category defined in box 308, and / or a fourth category defined in box 310. An example equation for the category penalty function as a weighted sum of penalty terms within categories defined in boxes 304-310 (and which can be used as the category penalty function for the “virtualization mode” category of rendering metadata in some implementations) is given by the following:

[0073] In some implementations, a category penalty function can be calculated that does not include penalties for mismatches between the rendering metadata type associated with the audio object and the rendering metadata type of the cluster to which the audio object is assigned or rendered. For example, such a category penalty function can be determined for the "bypass mode" category. In some implementations, such a category penalty function can be a weighted sum of a first in-category penalty determined at box 304, a second in-category penalty determined at box 306, and / or a third in-category penalty determined at box 308. An example equation for a category penalty function that is a weighted sum of the in-category penalties determined at boxes 304-308 (and in some implementations can be used as the category penalty function for the "bypass mode" category of rendering metadata) is given by:

[0074] It should be noted that the above-given category penalty function is used to calculate the category penalty function. E cat2 In the example, this category penalty function can be achieved by applying the penalty item within the fourth category. E G Set to 0 to apply the category penalty function E cat1 get.

[0075] It should be noted that the example category penalty function described above is merely illustrative. In some implementations, the category penalty function can be a weighted sum of any suitable in-category penalty terms, such as a weighted sum of in-category penalties for the first and second categories, a weighted sum of in-category penalties for the second and fourth categories, etc.

[0076] As mentioned above, for a given audio object associated with a specific rendering metadata category jThe vector of object-to-cluster gains, which indicates the audio object, can be determined by minimizing the category penalty function associated with the rendering metadata category. j The gain when rendered to one or more clusters (e.g., indicated as elements of a vector). For example, for an audio object associated with the “bypass mode” category of the rendering metadata, this can be achieved by minimizing the “bypass mode” category penalty function (e.g., in the equation above). E cat2 The object-to-cluster gain can be determined by minimizing the associated class penalty function. E To calculate audio objects j The gain vector is called For example, equations can be solved. ,in, E It is with audio objects j Category cost function for associated rendering metadata categories.

[0077] At 314, process 300 can compute an intra-category cost function for each render metadata category, based on the object-to-cluster gain of the audio objects associated with the render metadata category. In some implementations, the intra-category cost function can be determined based on the loudness of the audio objects within the render metadata category. Alternatively, in some implementations, it can be based on a corresponding intra-category penalty function (e.g., as described above). E cat1 and / or E cat2 (etc.) to determine the in-category cost function. Used to calculate the in-category penalty function. E An example equation for the cost function within a given category is given by the following:

[0078] In the equations given above, Represents an audio object jThe loudness of the audio object. It should be noted that the intra-category cost function may be based at least in part on any combination of the following: 1) the position of the audio object clusters relative to the position of the audio object assigned to the audio object cluster (e.g., based on the first intra-category penalty described above in box 304); 2) the left-right placement of the audio object relative to the left-right placement of the cluster to which the audio object has been assigned (e.g., based on the second intra-category penalty described above in box 306); 3) the distance of the audio object to the cluster to which the audio object has been assigned (e.g., based on the second intra-category penalty described above in box 306); 4) the loudness of the audio object (e.g., based on the third intra-category penalty described above in box 308); and / or 5) the similarity between the rendering metadata type associated with the audio object and the rendering metadata type associated with the cluster to which the audio object has been assigned (e.g., based on the fourth intra-category penalty described above in box 310).

[0079] In some implementations, the intra-category cost function can be determined as a loudness-weighted sum of location differences between audio objects and clusters. An example equation for calculating the intra-category cost function based on location differences is given below:

[0080] It should be noted that the cost function within each rendering metadata category can be determined. For example, the cost function within the first category. l 1 can be determined based on the "virtualization mode" category of the rendering metadata, and the cost function within the second category. l 2. This can be determined based on the "bypass mode" category of the rendering metadata. Similarly, when clustering audio objects for rendering in speaker rendering mode, the in-category cost function for region masking category, capture category, etc., can be calculated.

[0081] At 316, process 300 can compute a global cost function that combines category cost functions across different rendering metadata categories. For example, the global cost function can combine the first category cost function associated with the "virtualization mode" category of the rendering metadata (e.g., in the example given above). l 1) and a second category cost function associated with the “bypass mode” category of rendering metadata (e.g., in the example given above). l 2) Combining. Example equations used to calculate the global cost function (generally referred to in this paper as...) l global The following formula is given:

[0082] In the equations given above, aIt is a weighted constant that indicates the weight or importance of each rendering metadata category.

[0083] At 318, process 300 may reallocate clusters to rendering metadata categories, at least in part, based on the global cost function determined at box 316. For example, in some implementations, process 300 may do so by selecting the category that minimizes the global cost function. l global The number of clusters is used to reallocate clusters. As a more specific example, in some implementations, process 300 may select the number of clusters to be assigned to the first rendering metadata category as... m And the number of clusters to be assigned to the second rendering metadata category is selected as n .

[0084] In some implementations, the number of clusters to be assigned to a specific rendering metadata class in the current frame may differ from the number of clusters to be assigned to the same rendering metadata class in a previous frame (e.g., as a result of applying process 300 to a previous frame). In some implementations, the change in the number of clusters assigned to the current frame relative to the number assigned to a previous frame may be due to: a difference in the number of audio objects indicated in the current frame compared to the number indicated in a previous frame, a difference in the number of active audio objects indicated in the current frame compared to the number indicated in a previous frame, and / or a change in the spatial location of active audio objects across audio signal frames. As an example, in the current frame, the number of clusters to be assigned to a specific rendering metadata class in the current frame may differ from the number indicated in a previous frame. m Each cluster is assigned to the first rendering metadata category, while in previous frames... m’ Clusters are assigned to the first rendering metadata category. When adding two overlapping signals, including audio objects assigned to different rendering metadata categories, to the current frame, and when there are no free clusters available in the current frame for assignment to the first category, rendering may introduce artifacts. Adding additional clusters to a specific rendering metadata category by adding additional clusters that were not previously assigned to any rendering metadata category allows for more accurate clustering of audio objects assigned to that category without introducing rendering artifacts.

[0085] In some implementations, given in the previous frame m' Each cluster is assigned to the first rendering metadata category in that previous frame. n' Each cluster is assigned to the second rendering metadata category in the current frame. m Each cluster is assigned to the first rendering metadata category and in the current frame. n Clusters are assigned to the second rendering metadata category, and the increase of clusters in the first and second rendering metadata categories is given by the following formulas:

[0086] The number of clusters available for allocation to either the first category of rendering metadata or the second category of rendering metadata can be given by the following formula: m free = M total – ( m’ + n’ In some implementations, process 300 can be minimized. l global ( m , n To ensure that the following equation is satisfied, clusters are reassigned to the first and second rendering metadata categories: m + n ≤ M total and Δ m + Δ n ≤ m free 。 It should be noted that if cross-category assignment of audio objects is not permitted (e.g., assignment to a cluster associated with a rendering metadata category other than the rendering metadata category associated with the audio object), process 300 may reallocate clusters under this constraint.

[0087] For example, in M total The maximum number of clusters is 21 (e.g., a maximum of 21 clusters can be assigned to all rendering metadata categories) and m 'for 11 and n’ When the value is 10, m free It is 0, because m ' + n ' = M total Continuing with this example, process 300 can then be determined at box 318: m and n Neither can be increased because there are no clusters available for allocation. As a concrete example, if... m Set to 13 and n It is set to 8 (for example, to meet the standard). m + n ≤ M total ) ,but Δ m It is 2 and Δ n It is 0. However, due to Δ m + Δ n = 2, which is greater than m free (It is 0), therefore process 300 can determine that 13 is not valid for the current frame.m value.

[0088] It should be noted that although the example above describes two rendering metadata categories, the same technique can be applied to any suitable number of rendering metadata categories (e.g., three, four, etc.). For example, process 300 can be minimized. l global ( m i ), making and .

[0089] Then, process 300 can loop back to box 304. Process 300 can loop through boxes 304-318 until a stopping criterion is met. Examples of stopping criteria include determining that the minimum of the global cost function determined at box 316 has been reached, or that more iterations than a predetermined threshold have been performed through boxes 304-318. In some implementations, the allocation determined as a result of looping through boxes 304-318 until the stopping criterion is met can be called the "optimal allocation".

[0090] It should be noted that the frames of process 300 can be performed to determine the assignment of clusters to rendering metadata categories for a specific frame of the input audio signal. The frames of process 300 can be repeated for other frames of the input audio signal to determine the assignment of clusters to rendering metadata categories for those other frames. For example, in some implementations, the frames of process 300 can be repeated for each frame of the input audio signal, for every other frame of the input audio signal, etc.

[0091] Figure 4 An example of a process 400 for rendering audio objects to a cluster, according to some embodiments, is shown. The blocks of process 400 can be implemented on any suitable device, such as a server that generates an encoded audio signal based on audio objects included in the input audio signal. It should be noted that process 400 describes the process in general terms for a single frame of audio content; however, it should be understood that in some embodiments, the blocks of process 400 can be repeated for one or more other frames of the audio content, for example, to generate a complete output audio signal as a compressed version of the input audio signal. In some embodiments, one or more blocks of process 400 can be omitted. Additionally, in some embodiments, two or more blocks of process 400 can be executed substantially in parallel. The blocks of process 400 can be executed in any order, and are not limited to this. Figure 4 The order shown.

[0092] Process 400 may begin at 402, where cluster assignments to rendering metadata categories are obtained. For example, the assignment may indicate that a certain number of clusters are assigned to each rendering metadata category. As a more specific example, the assignment may indicate that a first number of clusters are assigned to a first rendering metadata category (e.g., the “Bypass Mode” category of rendering metadata) and a second number of clusters are assigned to a second rendering metadata category (e.g., the “Virtualization Mode” category of rendering metadata). In speaker rendering modes, other rendering metadata categories may include the “Snap” category of rendering metadata, the “Region Mask” category of rendering metadata, etc. In some implementations, cluster assignments may also indicate the centroid location of each cluster. In some implementations, the centroid location of each cluster may be used to calculate a penalty function used to determine the object-to-cluster gain at block 404.

[0093] In some implementations, the allocation of clusters to rendering metadata categories can be the result of an optimization process subject to various constraints or criteria (e.g., the maximum number of clusters) to determine the optimal allocation of clusters to rendering metadata categories. An example process for determining the allocation of clusters to rendering metadata categories is described in... Figure 3 The above is shown and combined Figure 3 Describe it.

[0094] It should be noted that cluster assignments to rendering metadata categories can be specified for each frame of the input audio signal. For example, the resulting assignment could indicate that, for the first frame of the input audio signal, clusters would be assigned to rendering metadata categories. m’ Each cluster is assigned to the first rendering metadata category, and for the second frame of the input audio signal, m Each cluster is assigned to the first rendering metadata category. The first and second frames of the input audio signal may or may not be consecutive frames.

[0095] At 404, process 400 can determine the object-to-cluster gain for each audio object in a frame of the input audio signal, assigned to a cluster of rendering metadata categories associated with that audio object. For example, in an audio object associated with the "bypass mode" category of rendering metadata and... m If a cluster has already been assigned to the "bypass mode" category of the rendering metadata, process 400 can determine whether the audio object is rendered in the "bypass mode" category assigned to the rendering metadata. m Object-to-cluster gain for individual clusters. It should be noted that the object-to-cluster gain for a specific audio object rendered to a particular cluster can be 0, indicating that the audio object has not been assigned to or rendered to that cluster.

[0096] In some implementations, process 400 can determine the object-to-cluster gain by individually minimizing the category penalty function for each rendering metadata category. It should be noted that determining the object-to-cluster gain by individually minimizing the penalty function for each rendering metadata category will prevent audio objects associated with a first rendering metadata category from being assigned or rendered to clusters assigned to a second rendering metadata category, where the first rendering metadata category is different from the second rendering metadata category. For example, in such an implementation, assigning and / or rendering audio objects associated with the “bypass mode” category of rendering metadata to clusters assigned to the “virtualization mode” category of rendering metadata will be prohibited. Examples of this clustering are shown in... Figure 1A The above is shown and combined Figure 1A Describe it.

[0097] In some implementations, the category penalty function can be combined with... Figure 3 The category penalty function described in box 312. For example, in conjunction with the iteration of the boxes in process 300, the category penalty function could be the final category penalty function determined for the final assignment when the stopping criterion is met. As a specific example, in the case of determining penalty items within four categories (e.g., in the case of headphone rendering mode, and for the "virtualization mode" category of rendering metadata), the category penalty function could be (as in conjunction with...) Figure 3 (as described in box 312)

[0098] As another concrete example, when determining the penalty within three categories (e.g., in the case of headphone rendering mode, and for the "bypass mode" category of rendering metadata), the category penalty function could be (as combined with...) Figure 3 (as described in box 312)

[0099] For example, in the case of headphone rendering mode, process 400 can determine a first set of object-to-cluster gains for a first set of audio objects associated with the "bypass mode" category by minimizing a first penalty function associated with the "bypass mode" category of the rendering metadata and for clusters assigned to the "bypass mode" category (e.g., as indicated in the assignment obtained at box 402). Continuing this example, process 400 can determine a second set of object-to-cluster gains for a second set of audio objects associated with the "virtualization mode" category by minimizing a second penalty function associated with the "virtualization mode" category of the rendering metadata and for clusters assigned to the "virtualization mode" category (e.g., as indicated in the assignment obtained at box 402).

[0100] Alternatively, in some implementations, process 400 can determine the object-to-cluster gain by minimizing a joint penalty function (e.g., considering all rendering metadata categories). In such an implementation, an audio object associated with a first rendering metadata category can be assigned or rendered to a cluster assigned to a second rendering metadata category, where the first rendering metadata category is different from the second rendering metadata category. For example, in such an implementation, an audio object associated with the “bypass mode” category of rendering metadata can be assigned and / or rendered to a cluster assigned to the “virtualization mode” category of rendering metadata. Examples of such clusters are... Figure 1B The figure is shown in the diagram and described above in conjunction with it.

[0101] The example equation representing the joint penalty function is:

[0102] In the equation above, E P , E D and E N These represent the first penalty item, the second penalty item, and the third penalty item described in boxes 304, 306, and 308, respectively. Therefore, E P , E D and E N The above combination can be used Figure 3 The techniques described in boxes 304, 306, and 308 are determined by considering audio objects and clusters across all rendering metadata categories. Similar to the techniques described above in conjunction with box 312, , , and This indicates the relative importance of each penalty term to the overall joint penalty function.

[0103] This indicates: 1) a penalty associated with a mismatch between assigning or rendering an audio object associated with the first category to a cluster assigned to the second rendering metadata category; and 2) a penalty associated with a mismatch between the rendering metadata type of the audio object and the rendering metadata type of the cluster to which the audio object is assigned or rendered (where the rendering metadata type of the audio object and the cluster are in the same rendering metadata category). For example, in the case of headphone rendering, This can indicate penalties for assigning and / or rendering audio objects associated with the "Bypass Mode" category of the rendering metadata to the "Virtualization Mode" category of the rendering metadata. Continuing this example, This may additionally or alternatively indicate a penalty for assigning audio objects associated with the "near" virtualization type to clusters primarily associated with the "medium" or "far" virtualization type. Used to determine... Example equations are given by the following:

[0104] In the equation above, U The following matrix indicates the relationship between the rendering mode and the target rendering mode. mode(j) Associated audio objects j Assign and / or render to the rendering mode mode(c) Penalty for associated clusters. For example, in the case of headphone rendering, examples of patterns (e.g., mode(j) and mode(c) Example values ​​(e.g., "bypass mode", "near", "medium", and "far") can include "bypass mode", "near" virtualization, "medium" virtualization, and "far" virtualization. In the case of headphone rendering, U It can be a 4 × 4 matrix, where rows indicate the patterns associated with the audio object, and columns indicate the patterns associated with the cluster to which the audio object is assigned or rendered. As a more specific example, in some implementations, U The first three rows and first three columns can correspond to different virtualization types (e.g., "near", "medium", and "far"), and U The fourth row and fourth column can correspond to the bypass mode. Such a matrix... U The example is:

[0105] As shown in the example matrix above U As shown, audio objects associated with the "bypass mode" category of rendering metadata are severely penalized when assigned to clusters assigned to the "virtualization mode" category of rendering metadata (e.g., U As indicated by the 1 in the last line. Similarly, audio objects associated with any type of the "virtualization mode" category of the rendering metadata (e.g., any of the "near," "medium," and / or "far" virtualization types) may be severely penalized when assigned to a cluster assigned to the "bypass mode" category of the rendering metadata (e.g., U(As indicated by the 1 in the last column). In other words, the penalty for cross-category assignment or rendering of audio objects is relatively greater compared to assigning or rendering audio objects to other rendering metadata types within the same rendering metadata category. For example, an audio object associated with the "near" virtualization type can be assigned to a cluster associated with the "medium" virtualization type, where the penalty is 0.3; it can be assigned to a cluster associated with the "far" virtualization type, where the penalty is 0.7; and it can be assigned to a cross-category cluster associated with "bypass mode" rendering metadata, where the penalty is 1.

[0106] At 406, process 400 may generate an output audio signal based on the object-to-cluster gain of each audio object (e.g., as determined at box 404). The output audio signal may include each audio object assigned to or rendered to one or more clusters based on the object-to-cluster gain determined for each audio object. This is used to generate a specific cluster. c The output audio signal (generally referred to in this article as) I out,c The example equation is:

[0107] As indicated in the equation above, the input audio signal I in,j The instructions in the middle j The audio object clusters are iterated over, and each is based on the object-to-cluster gain. Rendered to one or more clusters c .

[0108] It should be noted that the frame of process 400 can be repeated for one or more other frames of the input audio signal, such that the audio object indicated in those one or more other frames of the input audio signal is assigned or rendered to the respective clusters, thereby generating a complete output audio signal that includes multiple frames of the input audio signal (e.g., all frames of the input audio signal). In some embodiments, the complete output audio signal can be saved, transmitted to a device (e.g., a user device such as a mobile device, a television, a speaker, etc.) for presentation, etc.

[0109] Figure 5 This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure. As with the other figures provided herein, Figure 5The types and quantities of elements shown are provided by way of example only. Other embodiments may include more, fewer, and / or different types and quantities of elements. According to some examples, device 500 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 500 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

[0110] According to some alternative implementations, device 500 may be or may include a server. In some such examples, device 500 may be or may include an encoder. Thus, in some cases, device 500 may be a device configured for use in an audio environment such as a home audio environment; however, in other cases, device 500 may be a device configured for use in the “cloud,” such as a server.

[0111] In this example, device 500 includes an interface system 505 and a control system 510. In some embodiments, the interface system 505 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 505 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may be related to one or more software applications being executed by device 500.

[0112] In some implementations, the interface system 505 can be configured to receive or provide a content stream. The content stream may include audio data. Audio data may include, but is not limited to, audio signals. In some cases, audio data may include spatial data such as channel data and / or spatial metadata. In some examples, the content stream may include video data and corresponding audio data.

[0113] Interface system 505 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some embodiments, interface system 505 may include one or more wireless interfaces. Interface system 505 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 505 may include a control system 510 and a memory system (such as...). Figure 5The system 505 includes one or more interfaces between the optional memory system 515 shown in the diagram. However, in some cases, the control system 510 may include a memory system. In some embodiments, the interface system 505 may be configured to receive input from one or more microphones in the environment.

[0114] For example, the control system 510 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic and / or discrete hardware components.

[0115] In some implementations, the control system 510 may reside in more than one device. For example, in some implementations, a portion of the control system 510 may reside in a device within one of the environments depicted herein, and another portion of the control system 510 may reside in a device outside the environment, such as a server, mobile device (e.g., a smartphone or tablet computer), etc. In other examples, a portion of the control system 510 may reside in a device within an environment, and another portion of the control system 510 may reside in one or more other devices within the environment. For example, a portion of the control system 510 may reside in a device implementing a cloud-based service (such as a server), and another portion of the control system 510 may reside in another device implementing a cloud-based service (such as another server, storage device, etc.). In some examples, the interface system 505 may also reside in more than one device.

[0116] In some implementations, the control system 510 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 510 may be configured to implement a method for clustering audio objects.

[0117] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. For example, one or more non-transitory media may reside on... Figure 5 The optional memory system 515 and / or control system 510 shown are included. Therefore, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media on which software is stored. The software may, for example, include instructions for determining the allocation of clusters to various rendering metadata categories, assigning or rendering audio objects to allocated clusters, etc. The software may, for example, be provided by, for example, Figure 5 The control system 510 is executed by one or more components of the control system.

[0118] In some examples, device 500 may include Figure 5 The optional microphone system 520 is shown. The optional microphone system 520 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, the device 500 may not include the microphone system 520. However, in some such embodiments, the device 500 may still be configured to receive microphone data from one or more microphones in an audio environment via the interface system 510. In some such embodiments, a cloud-based implementation of the device 500 may be configured to receive microphone data or at least a noise metric corresponding to the microphone data from one or more microphones in an audio environment via the interface system 510.

[0119] According to some embodiments, the device 500 may include Figure 5 The optional loudspeaker system 525 is shown. The optional loudspeaker system 525 may include one or more loudspeakers, which may also be referred to herein as “loudspeakers” or more generally as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), device 500 may not include the loudspeaker system 525. In some implementations, device 500 may include headphones. The headphones may be connected to or coupled to device 500 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0120] Some aspects of this disclosure include a system or apparatus configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including input devices, memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0121] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on one or more audio signals, including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that may include input devices and memory) programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the invention are implemented as general-purpose processors or DSPs configured (e.g., programmed) to execute one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to execute one or more examples of the disclosed methods may be coupled to input devices (e.g., a mouse and / or keyboard), memory, and a display device.

[0122] Another aspect of this disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code (e.g., an encoder executable to perform one or more examples of the disclosed methods or steps) for performing the disclosed methods or steps.

[0123] While specific embodiments and applications of this disclosure have been described herein, it will be apparent to those skilled in the art that many changes can be made to the described embodiments and applications without departing from the scope of this disclosure as described and claimed herein. It should be understood that although certain forms of this disclosure have been shown and described, this disclosure is not limited to the specific embodiments described and shown or the specific methods described.

[0124] The enumerated example implementations are as follows: Example 1. A method for clustering audio objects, the method comprising: identifying a plurality of audio objects, wherein the audio objects are associated with metadata indicating spatial location information and rendering metadata; assigning audio objects from the plurality of audio objects to rendering metadata categories from a plurality of rendering metadata categories, wherein at least one rendering metadata category includes a plurality of rendering metadata types to be maintained; determining an allocation of a plurality of audio object clusters to each rendering metadata category, wherein the audio object cluster includes one or more audio objects from the plurality of audio objects having similar attributes; rendering the audio objects from the plurality of audio objects to the allocated plurality of audio object clusters based on the metadata indicating spatial location information and based on the assignment of the audio objects to the rendering metadata categories.

[0125] Example 2. The method as described in Example 1, wherein the rendering metadata category includes a bypass mode category and a virtualization category.

[0126] Example 3. The method as described in Example 2, wherein the plurality of rendering metadata types included in the virtualization category include a plurality of virtualization types, each representing the distance from the center of the head to the audio object.

[0127] Example 4. The method as described in Example 1, wherein the rendering metadata category includes either a region category or a capture category.

[0128] Example 5. The method of any one of Examples 1 to 4, wherein an audio object assigned to a first rendering metadata category is prohibited from being assigned to an audio object cluster in the plurality of audio object clusters that is assigned to a second rendering metadata category.

[0129] Example 6. The method of any one of Examples 1-5 further includes transmitting an audio signal, the audio signal including spatial information and gain information associated with each of the assigned plurality of audio object clusters, wherein the audio signal has less spatial distortion compared to an audio signal including spatial information and gain information associated with an audio object cluster in which an audio object assigned to the first rendering metadata category is assigned to an audio object cluster associated with the second rendering metadata category.

[0130] Example 7. The method of any one of Examples 1 to 6, wherein determining the allocation of the plurality of audio object clusters to each rendering metadata category comprises: (i) determining an initial allocation of the initial plurality of audio object clusters to each rendering metadata category; (ii) assigning the audio objects to the initial plurality of audio object clusters based on the metadata indicating spatial location information and based on the assignment of the audio objects to the rendering metadata category; (iii) determining, for each rendering metadata category, a category cost for assigning the audio objects to the initial plurality of audio object clusters; (iv) determining an updated allocation of the initial plurality of audio object clusters to each rendering metadata category based at least in part on the category cost of each rendering metadata category; and (iv) repeating (ii) to (iv) until a stopping criterion is met.

[0131] Example 8. The method as described in Example 7, wherein determining the category cost of assigning the audio object to the initial plurality of audio object clusters is based on the position of the audio object clusters assigned to the rendering metadata category and the position of the audio object assigned to the audio object clusters assigned to the rendering metadata category.

[0132] Example 9. The method as described in Example 8, wherein the category cost is based on the left-right placement of the audio object relative to the left-right placement of the audio object cluster to which the audio object has been assigned.

[0133] Example 10. The method of any one of Examples 7 to 9, wherein determining the category cost of assigning the audio object to the initial plurality of audio object clusters is based on the loudness of the audio object.

[0134] Example 11. The method of any one of Examples 7 to 10, wherein determining the category cost of assigning the audio object to the initial plurality of audio object clusters is based on the distance of the audio object to the audio object cluster to which the audio object has been assigned.

[0135] Example 12. The method of any one of Examples 7 to 11, wherein determining the category cost of assigning the audio object to the initial plurality of audio object clusters is based on the similarity between the rendering metadata type of the audio object and the rendering metadata type of the audio object cluster to which the audio object has been assigned.

[0136] Example 13. The method of any one of Examples 7 to 12, further comprising determining a global cost based on the category cost of each rendering metadata category, wherein the update allocation of the initial plurality of audio object clusters is based on the global cost.

[0137] Example 14. The method as described in Example 12, wherein repeating (ii) to (iv) until the stopping criterion is reached includes determining the minimum value of the global cost that has been achieved.

[0138] Example 15. The method of any one of Examples 7 to 14, wherein determining the update allocation includes changing the number of audio object clusters allocated to at least one of the plurality of rendering metadata categories.

[0139] Example 16. The method of Example 15 further includes determining a global cost based on the category cost of each rendering metadata category, wherein the number of audio object clusters is determined based on the global cost.

[0140] Example 17. The method as described in Example 16, wherein determining the number of audio object clusters includes minimizing the global cost by being subject to a constraint on the number of audio object clusters that indicates the maximum number of audio object clusters that can be added.

[0141] Example 18. The method of any one of Examples 1 to 17, wherein rendering an audio object from the plurality of audio objects to the assigned plurality of audio object clusters comprises: determining the object-to-cluster gain of each audio object from the plurality of audio objects when rendered to one or more audio object clusters assigned to the rendering metadata category to which the audio object is assigned.

[0142] Example 19. The method as described in Example 18, wherein the object-to-cluster gain of audio objects assigned to a first category of the plurality of rendering metadata categories is determined independently of the object-to-cluster gain of audio objects assigned to a second category of the plurality of rendering metadata categories.

[0143] Example 20. The method as described in Example 18, wherein the object-to-cluster gain of audio objects assigned to a first category of the plurality of rendering metadata categories is jointly determined with the object-to-cluster gain of audio objects assigned to a second category of the plurality of rendering metadata categories.

[0144] Example 21. The method of any one of Examples 1 to 20 further includes transmitting an audio signal, the audio signal including spatial information and gain information associated with each of the assigned plurality of audio object clusters, wherein transmitting the audio signal requires less bandwidth than transmitting an audio signal including spatial information and gain information associated with each of the plurality of audio objects.

[0145] Example 22. An apparatus configured to perform the method as described in any one of Examples 1 to 21.

[0146] Example 23. A system configured to implement the method as described in any one of Examples 1 to 21.

[0147] Example 24. One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method as described in any one of Examples 1 to 21.

Claims

1. A method for clustering audio objects, the method comprising: Identify multiple audio objects, wherein each audio object is associated with corresponding metadata and corresponding rendering metadata that indicate corresponding spatial location information; The audio objects among the plurality of audio objects are assigned to a rendering metadata category among a plurality of rendering metadata categories, wherein at least one rendering metadata category includes a plurality of rendering metadata types to be maintained; Determine the assignment of multiple audio object clusters to each rendering metadata category, wherein the audio object cluster includes one or more audio objects with similar attributes among the multiple audio objects; Based on the metadata indicating spatial location information and based on the assignment of the audio objects to the rendering metadata category, the audio objects among the plurality of audio objects are rendered into the assigned plurality of audio object clusters.

2. The method as described in claim 1, wherein, The rendering metadata categories include bypass mode category and virtualization category.

3. The method as described in claim 2, wherein, The plurality of rendering metadata types included in the virtualization category include a plurality of virtualization types, each of which represents the distance from the center of the head to the audio object.

4. The method of claim 1, wherein, The rendering metadata categories include either the region category or the capture category.

5. The method according to any one of claims 1 to 4, wherein, Audio objects assigned to the first rendering metadata category are prohibited from being assigned to audio object clusters in the multiple audio object clusters that are assigned to the second rendering metadata category.

6. The method of any one of claims 1 to 4, further comprising transmitting an audio signal, the audio signal including spatial information and gain information associated with each of the assigned plurality of audio object clusters, wherein, The audio signal has less spatial distortion compared to an audio signal that includes spatial information and gain information associated with an audio object cluster in which audio objects assigned to a first rendering metadata category are assigned to an audio object cluster associated with a second rendering metadata category.

7. The method according to any one of claims 1 to 4, wherein, Determining the allocation of the plurality of audio object clusters to each rendering metadata category includes: (i) Determine the initial allocation of the initial multiple audio object clusters to each rendering metadata category; (ii) Assign the audio object to the initial plurality of audio object clusters based on the metadata indicating spatial location information and based on the assignment of the audio object to the rendering metadata category; (iii) For each rendering metadata category, determine the category cost of assigning the audio object to the initial plurality of audio object clusters; (iv) Determine the update allocation of the initial plurality of audio object clusters to each rendering metadata category based at least in part on the category cost of each rendering metadata category; and (v) Repeat (ii) to (iv) until the stopping criterion is met.

8. The method of claim 7, wherein, The category cost for assigning the audio object to the initial plurality of audio object clusters is determined based on the location of the audio object clusters assigned to the rendering metadata category and the location of the audio object assigned to the audio object clusters assigned to the rendering metadata category.

9. The method of claim 8, wherein, The category cost is based on the left-right placement of the audio object relative to the left-right placement of the audio object cluster to which the audio object has been assigned.

10. The method of claim 7, wherein, The category cost for assigning the audio object to the initial plurality of audio object clusters is determined based on the loudness of the audio object.

11. The method of claim 7, wherein, The category cost for assigning the audio object to the initial plurality of audio object clusters is determined based on the distance from the audio object to the audio object cluster to which the audio object has been assigned.

12. The method of claim 7, wherein, The category cost of assigning the audio object to the initial plurality of audio object clusters is determined based on the similarity between the rendering metadata type of the audio object and the rendering metadata type of the audio object cluster to which the audio object has been assigned.

13. The method of claim 7, further comprising determining a global cost based on the category cost of each rendering metadata category, wherein, The update allocation of the initial multiple audio object clusters is based on the global cost.

14. The method of claim 13, wherein, Repeat steps (ii) through (iv) until the stopping criterion is reached, including determining the minimum global cost that has been achieved.

15. The method of claim 7, wherein, Determining the update allocation includes changing the number of audio object clusters assigned to at least one of the plurality of rendering metadata categories.

16. The method of claim 15, further comprising determining a global cost based on the category cost of each rendering metadata category, wherein, The number of audio object clusters is determined based on the global cost.

17. The method of claim 16, wherein, Determining the number of audio object clusters includes minimizing the global cost by being subject to a constraint on the number of audio object clusters that indicates the maximum number of audio object clusters that can be added.

18. The method according to any one of claims 1 to 4, wherein, Rendering an audio object from the plurality of audio objects to the assigned plurality of audio object clusters includes: determining the object-to-cluster gain of each audio object from the plurality of audio objects when rendered to one or more audio object clusters assigned to the rendering metadata category to which the audio object is assigned.

19. The method of claim 18, wherein, The object-to-cluster gain of audio objects assigned to the first category of the plurality of rendering metadata categories is determined independently of the object-to-cluster gain of audio objects assigned to the second category of the plurality of rendering metadata categories.

20. The method of claim 18, wherein, The object-to-cluster gain of audio objects assigned to the first category of the plurality of rendering metadata categories is determined jointly with the object-to-cluster gain of audio objects assigned to the second category of the plurality of rendering metadata categories.

21. The method of any one of claims 1 to 4, further comprising transmitting an audio signal, the audio signal including spatial information and gain information associated with each of the assigned plurality of audio object clusters, wherein, Transmitting the audio signal requires less bandwidth compared to transmitting an audio signal that includes spatial and gain information associated with each of the plurality of audio objects.

22. An apparatus for clustering audio objects, the apparatus being configured to implement the method as claimed in any one of claims 1 to 21.

23. An electronic system configured to implement the method as claimed in any one of claims 1 to 21.

24. One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method as described in any one of claims 1 to 21.

25. A computer program product comprising a computer program, said computer program including instructions for controlling one or more devices to perform the method as described in any one of claims 1 to 21.

Images