Method and apparatus for efficient audio rendering

Abstract

A method for rendering audio sources is disclosed, the method comprising: determining, for each of a plurality of audio sources, a rendering control value based on one or more parameters indicative of the perceptual relevance of each audio source; comparing, for each of the plurality of audio sources, the rendering control value to a perceptibility threshold to determine whether each audio source satisfies a perceptual irrelevance criterion; modifying audio sources of the plurality of audio sources whose rendering control values ​​satisfy the threshold to obtain modified audio sources; and rendering unmodified audio sources of the plurality of audio sources against the modified audio sources. Further, methods, apparatus, and computer program products for processing audio are disclosed.

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 431,822, filed December 12, 2022, and U.S. Provisional Application No. 63 / 491,258, filed March 20, 2023, the contents of which are incorporated herein by reference in their entireties.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to methods for rendering audio sources, and more particularly to controlling the rendering of audio sources for efficient audio rendering.

[0003] Although some embodiments are described herein with particular reference to that disclosure, it should be understood that the disclosure is not limited to such applications but is applicable in a broader context. [Background technology]

[0004] Any discussion of background art throughout this disclosure should not be taken as an admission that such art is widely known or forms common general knowledge in the art.

[0005] Audio rendering, especially for high-quality immersive content, is computationally expensive. Rendering immersive audio within a reasonable operating time, especially on power-constrained devices, only allows for very complex and limited numerical operations on the processors they contain. Consequently, the output audio is often of poor quality, resulting in a poor listener experience.

[0006] Therefore, a need exists for a method and apparatus that allows for efficient rendering of an audio source without degrading the perceptual quality of the resulting rendered audio output. Summary of the Invention

[0007] According to a first aspect of the present disclosure, a method for rendering audio sources is provided. The method includes determining, for each of a plurality of audio sources, a rendering control value based on one or more parameters indicative of the perceptual relevance of each audio source. The method further includes comparing, for each of the plurality of audio sources, the rendering control value to a perceptibility threshold to determine whether each audio source satisfies a perceptual irrelevance criterion. The method further includes modifying audio sources of the plurality of audio sources whose rendering control values ​​satisfy the threshold to obtain modified audio sources. The method then includes rendering unmodified audio sources of the plurality of audio sources against the modified audio sources.

[0008] In some embodiments, the one or more parameters may include a loudness of each audio source, and the rendering control value may then be determined based on the loudness value.

[0009] In some embodiments, the one or more parameters may include a relationship between the loudness of a single audio source and the loudness of an audio output obtained by rendering multiple audio sources excluding the single audio source.

[0010] In some embodiments, the one or more parameters may include a relationship between the short-term spectral energy of a single audio source and the spectral energy of an audio output obtained by rendering multiple audio sources.

[0011] In some embodiments, the rendered audio output may be associated with a psychoacoustic masking model.

[0012] In some embodiments, modifying the audio sources for which the rendering control values ​​satisfy a threshold may include culling the audio sources, where the modified audio sources may then correspond to a culled subset of the plurality of audio sources.

[0013] In some embodiments, rendering unmodified audio sources relative to modified audio sources of the plurality of audio sources may include not rendering audio sources included in a culling subset.

[0014] In some embodiments, the one or more parameters may include the positional proximity of the audio source to the listener position.

[0015] In some embodiments, the rendering control value may be determined based on the distance of each audio source position of the audio source for which the rendering control value is determined relative to another audio source, and the rendering control value may be compared to a portion of the distance between the listener position and the nearest source position.

[0016] In some embodiments, the one or more parameters may include directional proximity with respect to the listener position.

[0017] In some embodiments, the rendering control values ​​may be determined based on at least one of an azimuth angle or an elevation angle between each audio source position for which the rendering control values ​​are determined and the listener position.

[0018] In some embodiments, modifying the audio sources may include clustering audio sources whose rendering control values ​​satisfy a threshold. The modified audio sources may then correspond to the clustered audio sources.

[0019] In some embodiments, rendering unmodified audio sources to modified audio sources of the plurality of audio sources may include replacing cluster audio sources with fewer audio sources and rendering the fewer audio sources as modified audio sources.

[0020] In some embodiments, the one or more parameters may include one or more of the following: acoustic occlusion, optical occlusion, or abstraction. A rendering control value may then be determined based on the level of occlusion or abstraction for each audio source.

[0021] In some embodiments, modifying the audio source may include converting the type of the audio source whose rendering control value satisfies a threshold. The modified audio source may then correspond to the typecast audio source.

[0022] In some embodiments, rendering unmodified audio sources relative to modified audio sources of the plurality of audio sources may include rendering typecast audio sources as modified audio sources.

[0023] In some embodiments, for an audio source of a plurality of audio sources that has a directional pattern, if the rendering control value meets a threshold, no directivity calculation may be performed.

[0024] In some embodiments, the method may further include receiving control information indicating whether to perform modification of some or all of the audio sources whose rendering control values ​​satisfy a threshold.

[0025] In some embodiments, the control information may be one of activation or deactivation parameters, grouping data, prioritization information, and / or visibility information.

[0026] According to a second aspect of the present disclosure, there is provided an apparatus for rendering an audio source, the apparatus including one or more processors configured to perform a method, the method comprising: determining, for each of a plurality of audio sources, a rendering control value based on one or more parameters indicative of the perceptual relevance of each audio source; for each of a plurality of audio sources, comparing the rendering control value to a perceptibility threshold to determine whether each audio source meets a perceptual irrelevance criterion; modifying an audio source among the plurality of audio sources whose rendering control value satisfies a threshold to obtain a modified audio source; Rendering unmodified audio sources against modified audio sources of a plurality of audio sources; Includes.

[0027] According to a third aspect of the present disclosure, there is provided an apparatus, the apparatus including: a processor; and a memory coupled to the processor, the memory storing instructions for the processor, the processor adapted to perform the method described herein.

[0028] According to a fourth aspect of the present disclosure, there is provided a program comprising instructions that, when executed by a processor, cause the processor to perform the method described herein. A computer-readable storage medium may store the program.

[0029] It will be understood that device (system) features and method steps may be interchanged in many ways. In particular, details of the disclosed methods can be implemented by corresponding devices (systems), and vice versa, as will be understood by those skilled in the art. Furthermore, it will be understood that statements made above with respect to methods apply equally to corresponding devices (systems), and vice versa. [Brief explanation of the drawings]

[0030] Exemplary embodiments of the present disclosure are now described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates an example of a method for rendering an audio source according to one embodiment of the present disclosure. [Figure 2] 10A and 10B illustrate schematic examples of audio source culling according to an embodiment of the present disclosure. [Figure 3] 1 illustrates a schematic example of clustering of audio sources according to one embodiment of the present disclosure. [Figure 4] 10 illustrates a schematic diagram of another example of clustering of audio sources according to an embodiment of the present disclosure. [Figure 5] 10A and 10B illustrate schematic examples of typecasting of audio sources according to one embodiment of the present disclosure. [Figure 6] 1 illustrates an example method for processing audio according to one embodiment of the present disclosure. [Figure 7] 1 illustrates another example of a method for processing audio according to an embodiment of the present disclosure. [Figure 8] 10 illustrates yet another example of a method for processing audio according to an embodiment of the present disclosure. [Figure 9] 10 illustrates an example of the use of control information according to one embodiment of the present disclosure. [Figure 10] 10A and 10B illustrate schematic examples of control information parameters according to an embodiment of the present disclosure. [Figure 11] 1 illustrates an example of an apparatus including a processor and a memory coupled to the processor, according to one embodiment of the present disclosure.

[0031] It should be noted that, wherever practicable, like or similar reference numerals may be used in the figures and may indicate like or similar functionality. The figures depict embodiments of the disclosed apparatus (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0032] Furthermore, in the figures, when a connecting element, such as a solid or dashed line or arrow, is used to describe a connection, relationship, or association between or within two or more other schematic elements, the absence of any such connecting element does not mean that the connection, relationship, or association is absent. In other words, some connections, relationships, or associations between elements may not be shown in the figures so as not to obscure the present disclosure. Furthermore, for ease of explanation, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, when a connecting element represents communication of signals, data, or instructions, it should be understood by those skilled in the art that such element may affect communication via one or more signal paths as needed. DETAILED DESCRIPTION OF THE INVENTION

[0033] <Summary> The methods and apparatus described herein provide technical benefits and advantages of reducing computational and memory system workload without degradation of the perceptual quality of the resulting rendered audio output. Absence of degradation of perceptual quality may be understood as providing a similar (or better) listener subjective experience. For example, the present disclosure provides methods for modifying audio output to achieve this benefit, as opposed to unmodified audio output. Furthermore, the present disclosure provides the further advantage of workload reduction. This reduction may be achieved by modifying and / or deactivating digital signal processing (DSP) of selected audio sources, instances, and optionally associated metadata processing.

[0034] The parameters, downmix matrices, and / or related information of the methods described herein can be defined and signaled by the encoder, decoder / renderer, and / or application. For example, the encoder can operate in an "encoder-assisted" mode, which allows a content creator to manually influence an otherwise automated method. This can provide the advantage of avoiding pointless culling, as described below. Alternatively, this information can be provided by the decoder and / or renderer (e.g., automatically in a "default" mode). Alternatively, this information can be provided by the application (e.g., automatically provided by an application or system middleware in a "system-assisted" mode).

[0035] How to Render an Audio Source The present disclosure aims to reduce the number of rendered audio sources in order to remove complexity in the renderer without psychoacoustic effects. Such audio sources may be any type of audio input source, such as channel (static) sources, audio objects, Higher Order Ambisonic(s) (HOA), First Order Ambisonics (FOA), and / or B-Format Ambisonics, for example.

[0036] Referring to the example of FIG. 1, a method 100 for rendering audio is shown.

[0037] In step S101 of the method, for each of a plurality of audio sources, a rendering control value is determined based on one or more parameters indicative of the perceptual relevance of each audio source.

[0038] In step S102 of the method, for each of the plurality of audio sources, the rendering control value is compared to a perceptibility threshold to determine whether each audio source meets the perceptual irrelevance criterion.

[0039] In step S103 of the method, an audio source among the plurality of audio sources whose rendering control value satisfies a threshold is modified to obtain a modified audio source.

[0040] Then, in S104, the unmodified audio sources are rendered relative to the modified audio sources among the plurality of audio sources.

[0041] In one embodiment, the one or more parameters (for determining the rendering control value) may include the loudness of each (single) audio source. The rendering control value may then be determined based on each loudness value. Depending on the use case, the loudness value may be derived from: 1) Metadata from existing audio standards set by the MPEG Audio Group, including MPEG-H 3D Audio (ISO_IEC_23008-3) or MPEG-D Part 4 (Dynamic Range Control). For example, metadata from legacy content may be presented. 2) Information from an encoder established by the MPEG Audio Group, such as an encoder compatible with the MPEG-H 3D Audio or MPEG-I Audio standards. For example, loudness information may be estimated and transmitted via the MPEG-H Audio Stream (MHAS) packet payload. 3) Information from the MPEG-I renderer, e.g., estimated in real time using the audio content and rendering gain.

[0042] The above mentioned loudness data sources can be applied alone or in combination, for example in the case of content without or with unreliable loudness information, social virtual reality (VR) content, etc.

[0043] In general, loudness can be signaled and processed in different ways according to different loudness definitions and measurement methods, eg, short-term, instantaneous, level-controlled, IBU-defined, etc.

[0044] Alternatively or additionally, in one embodiment, the one or more parameters may include a relationship between the loudness of a single audio source and the loudness of an audio output obtained by rendering multiple audio sources excluding the single audio source. Such a relationship may be, for example, without limitation, a signal-to-noise ratio. Other measures may be considered as well, for example, using spatial masking.

[0045] Alternatively or additionally, in one embodiment, the one or more parameters may include a relationship between the short-term spectral energy of a single audio source and the spectral energy of an audio output obtained by rendering multiple audio sources. The rendered audio output may be associated with a psychoacoustic masking model.

[0046] The above-mentioned parameters may be used alone or in combination to determine a respective rendering control value for each of a plurality of audio sources. Thus, the rendering control value may be said to indicate the perceptual relevance (perceptibility) of each of the plurality of audio sources. Then, by comparing the rendering control value for each of the plurality of audio sources with a (predetermined) perceptibility threshold, it is possible to indicate whether each of the audio sources is perceptually relevant or irrelevant, i.e., whether each of the audio sources satisfies the perceptual irrelevance criterion. For example, if the rendering control value is equal to or greater than a certain threshold, each of the audio sources may be considered perceptually relevant. If the rendering control value is less than a certain threshold, each of the audio sources may be considered perceptually irrelevant, i.e., each of the audio sources meets / satisfies the perceptual irrelevance criterion.

[0047] For example, an audio source associated with birdsong may become perceptually irrelevant if a listener begins to experience the noise of heavy rain, in which case the rendering control values ​​meet the respective (perceptual relevance) thresholds and each audio source meets the perceptual irrelevance criterion.

[0048] In one embodiment, modifying audio sources whose rendering control values ​​satisfy a threshold may include culling the audio sources. The modified audio sources may then correspond to culling subsets of the multiple audio sources. Referring to the example of FIG. 2, the culling of audio sources is schematically illustrated by forming respective culling subsets. FIG. 2 illustrates multiple audio sources 201, 202 associated with respective audio scenes 200. Referring again to the above example, audio source 202 may be associated with birdsong, and audio source 201 may be associated with light rain. If a listener now begins to experience the noise of heavy rain 201, the audio source associated with birdsong 202 may become perceptually irrelevant, i.e., the rendering control values ​​may satisfy the perceptibility threshold. In this case, audio source 202 is modified to enable culling of the audio source. The culled audio sources may then correspond to respective culling subsets 203.

[0049] In this context, culling can be said to refer to removing each perceptually irrelevant audio source, i.e., in one embodiment, rendering unmodified audio sources (whose rendering control values ​​do not satisfy the perceptibility threshold) of a plurality of audio sources relative to a modified audio source may include not rendering the audio sources included in the culling subset.

[0050] By considering additional perceptual energy / loudness-related aspects for perceptual relevance estimation and setting each rendering control condition based on comparing each rendering control value with a perceptibility (perceptual relevance) threshold to determine whether each audio source meets the perceptual irrelevance criterion, an improvement over conventional methods for detecting audio source irrelevance is achieved, in that conventional methods only consider the distance (from the listener to the audio object) and / or the rendering gain (of the audio object). In other words, conventional methods can be said to apply blind culling in that they do not consider the actual signal and signal energy. Therefore, conventional methods cannot detect a subclass of perceptually irrelevant audio sources, i.e., sources that are not audible but whose corresponding distance and gain values ​​exceed the conventional culling thresholds. Therefore, conventional methods may remove audio sources that should be kept, and vice versa.

[0051] In one embodiment, the one or more parameters (for determining the rendering control value) may alternatively or additionally include the positional proximity (closeness of source locations) of the audio source relative to the listener location. The rendering control value may then be determined based on the distance of each audio source location relative to another audio source for which a rendering control value is determined, and the rendering control value may be compared to a portion of the distance between the listener location and the nearest source location. For example, the distance between N audio source locations may be calculated as T of the distance between the listener and the nearest source. distance % can be made smaller than T distance % represents a variable that indicates a percentage value (e.g., 5%). When a listener explores a scene containing multiple audio sources (i.e., the original N audio sources), the distance between the listener and the closest source can be calculated. This distance to the closest source can be represented as a variable Tc. Additionally, the distance between each remaining audio source in the scene can also be calculated. Each of these distances is compared against Tc. If the distance between two sources is Tc, then Tc is distance%, then these sources are clustered (grouped together). Since the comparison is done across all audio sources, the cluster can add or remove more members (audio sources).

[0052] Figure 3 shows an example of an abstract visualization of clustering based on locational proximity. As shown in Figure 3, cluster A 301 contains audio sources 301A that are closest to each other in terms of distance. Cluster B 302 contains different sources 302B that are also closest to each other in terms of distance. Conversely, source X 303 is located further away from audio sources 301A, 302B in both clusters A 301 and B 302, and therefore is not assigned to either cluster.

[0053] Alternatively or additionally, the one or more parameters may include directional proximity (close azimuth and elevation angles of the sources) with respect to the listener position. The rendering control values ​​may then be determined based on at least one of the azimuth angles or elevation angles between each audio source position of the audio sources for which the rendering control values ​​are determined and the listener position. For example, the azimuth angles and elevation angles associated with the listener position for the N audio sources may each be T azimuth Degree and T elevation 4 shows an example of an abstract visualization of clustering based on directional and / or positional / distance proximity. As can be seen in FIG. 4, the angular / directional difference in terms of azimuth or elevation between any two sources (e.g., A2 and An) as seen by listener L is T azimuth Degree and T elevation Assume that the distance between the sources is less than 100 degrees. Then, these sources are clustered together. azimuth Degree and T elevation The threshold may be, for example, 5 degrees. It may also be assumed that the distance between these audio sources (A, A1, A2, A3, and An) is small, so they are clustered based on proximity of location or distance.

[0054] The above-mentioned parameters may be used alone or in combination to determine each rendering control value for each of the multiple audio sources. Therefore, the rendering control value may be said to indicate the perceptual relevance (perceptibility) of each audio source. Then, for each of the multiple audio sources, setting the rendering control value to a (predetermined) perceptibility threshold may indicate whether each audio source is perceptually relevant, i.e., whether each audio source satisfies the perceptual irrelevance criterion. For example, if the rendering control value is equal to or greater than a certain threshold, each audio source may be considered perceptually relevant. If the rendering control value is less than a certain threshold, each audio source may be considered perceptually irrelevant, i.e., each audio source meets / satisfies the perceptual irrelevance criterion. Alternatively, or in addition to the above-mentioned loudness-related parameters, the above-mentioned parameters represent perceptual localization-related aspects for perceptual relevance / irrelevance estimation.

[0055] In one embodiment, changing an audio source may include clustering audio sources whose rendering control values meet a perceptibility threshold. Then, the changed audio source may correspond to the clustered audio source. In one embodiment, rendering an unchanged audio source (where the rendering control value may not meet the perceptibility threshold) for a changed audio source among a plurality of audio sources may include replacing the clustered audio source with a smaller number of audio sources and rendering the smaller number of audio sources as the changed audio source. For example, N audio sources may be replaced by one or more audio sources M (M < N, preferably M = 1) including a weighted downmix signal or a multi-channel audio signal. That is, assuming that a total of N audio sources are clustered, these sources may be replaced by M audio sources. M may be the number of changed audio sources rendered instead of N. Thus, the M audio sources still represent the same cluster.

[0056] By clustering audio sources whose rendering control values meet the threshold, advantageously, some audio sources can be replaced with a smaller number of changed audio sources, thereby reducing the number of audio sources to be rendered.

[0057] Alternatively or additionally, in one embodiment, one or more parameters may include one or more of acoustic occlusion, optical occlusion, or abstraction later. Then, the rendering control value may be determined based on the level of occlusion or abstraction of each audio source.

[0058] In one embodiment, modifying audio sources may include converting the type of audio source whose rendering control values ​​satisfy a threshold. That is, modifying audio sources may include converting each audio source from one type to another. The resulting type can be rendered more computationally efficient. This is illustrated schematically in the example of FIG. 5, which shows multiple audio sources 501 associated with each audio scene 500. For example, if the distance 502 between the listener L and some of the audio sources 501 is large enough, those audio sources 501 (e.g., the diffuse sound of a waterfall) can be replaced by a single audio object 503 (e.g., a point source of a distant waterfall). Or, many audio point sources (e.g., raindrops) can be replaced by a single Higher Order Ambisonics (HOA) (e.g., rain) signal.

[0059] The modified audio sources may then correspond to the typecast audio sources. The parameters for determining each rendering control condition (rendering control value compared to a threshold) for typecasting may depend, among other things, on positional and / or directional proximity and / or level of acoustic / optical occlusion / abstraction, as described above. For example, if several audio sources are occluded and acoustically significant, they can simply be replaced with a single point-source audio or FOA signal.

[0060] Rendering an unmodified audio source (for which the rendering control value does not meet the perceptibility threshold) to a modified audio source among a plurality of audio sources may therefore include rendering the typecast audio source as the modified audio source.

[0061] Further to the above, in one embodiment, for an audio source among a plurality of audio sources that has a directional pattern, if the rendering control value satisfies a threshold, then no directivity calculation may be performed.

[0062] An example may be the directional sound of a car engine. Assuming a car is traveling on a circular track and a listener is standing at a particular location / stage, such as in a racing car scenario, the effect of the directional engine sound will be perceived as the car passes by the listener. However, if the car is far away, the listener will only hear a faint engine sound, so the directional effect is negligible and perceptually irrelevant, and so no directionality calculations need be performed. In some embodiments, a source may also be considered for inclusion in the culling subset if there is a significant obstacle / occlusion in the middle of the circular track, such as a hill or a solid block of rock.

[0063] In one embodiment, directivity calculations can be skipped by additionally considering a so-called worst-case directional gain estimation parameter. This worst-case estimate is easily calculated, for example, by simply taking the maximum directional gain within a given directional pattern of an audio source. This estimate can be used in combination with other existing rendering stage thresholds, for example, a 60 dB culling threshold, so that if the audio source rendering gain meets the "culling" stage gain threshold, the directivity calculation / stage can be skipped.

[0064] In one embodiment, the method may further include receiving control information indicating whether to perform modification of some or all of the audio sources whose rendering control values ​​satisfy a threshold.

[0065] The following parameters can be used to control the application of the rendering stages of perceptual audio culling, clustering and typecasting: (De)activation parameters can be included in the control information. Such parameters make it possible, for example, to disable the modification of the narrator's voice signal even if the respective audio source has a rendering control value that meets a threshold, i.e., meets the perceptual irrelevance criterion. Alternatively or additionally, grouping data for culling / clustering / typecasting can be included in the control information. Such grouping data can include, for example, means to define a set of audio sources within a particular region (e.g., a section emitting car sounds) or a particular signal category (e.g., ambient sounds of rain and wind sounds). Alternatively or additionally, for each group, the encoder can define culling and clustering strategies to combine or remove objects, which can be included in the control information. For example, If all objects in a group are below a certain threshold loudness, the group may be combined into one object (loudness-based culling); Groups may be combined into one object if the azimuth / elevation angles of all objects are within a certain range (orientation-based clustering). If one or many objects in a group are acoustically shielded, the group may be combined into one object.

[0066] Alternatively or additionally, the control information may include a prioritization of the perceptual relevance of the signals (and their informational importance) to the listener (and the focus of the listener's attention). For example, to improve speech intelligibility, culling of irrelevant audio sources may be applied (instead of leveling / EQing them) for a "cocktail party effect" reproduction. Alternatively or additionally, the control information may include the visibility of the graphical representation of the audio sources to the listener (i.e., whether a visual representation of the audio source is visible to the user). For example, audio sources may be associated with visual representations such as: The video is not within the rendered viewport (behind a listener), Obstructed by optically non-transparent obstructions.

[0067] Such audio sources may be treated differently depending on their nature and application scenario: for example, the thresholds for positional or directional proximity conditions may differ depending on the visibility of the virtual object.

[0068] To avoid frequent "switching" between the culling, clustering and type-casting rendering stages, the following aspects can be further used: hysteresis (dependence of the current decision on the history of previous decisions); Distance relative to the listener's position (and its change over time - position velocity), The direction (and its change over time - angular velocity) relative to the listener's position, for example a time threshold, can be determined based on the measured velocity.

[0069] All parameters and / or downmix matrices of the above aspects can be defined and signaled as follows: Encoder-assisted mode (manual by content creator), which allows content creators to influence the renderer's automatic culling (e.g., avoid pointless culling), Renderer - "default" mode (automatic), Application - "System assisted" mode (automatically) by application or system middleware.

[0070] In particular, parameters for all rendering stages can be defined for each (virtual) object. All these rendering stages can be implemented as part of the MPEG-I rendering process or as MPEG-I audio "external tools." The transition from the "culling-," "clustering-," and "typecast-" states to the original state is initiated (when the corresponding condition is met) and the sound source energy is low. That is, in another embodiment, the above method can be implemented by a respective two-stage process for determining the "correct" frame for applying the complexity reduction measures. The two steps include, in a first step, determining that the complexity reduction measures (i.e., modifying the audio source as described) should be applied, and in a second step, determining the time or duration for applying each measure.

[0071] The above-described methods may be implemented by respective devices including one or more processors. Alternatively or additionally, the above-described methods may be implemented in the form of respective programs including instructions that, when executed by a processor, cause the processor to perform the method. The programs may be stored in a computer-readable storage medium.

[0072] The above-described aspects of controlling the rendering of multiple audio sources may be implemented alone or in combination within a single method, while the above-described aspects of reducing complexity may alternatively be implemented alone or in combination, or by each individual method of processing audio described below.

[0073] Audio Source Culling Methods according to the present disclosure are directed to such audio source selection (or "culling").

[0074] The present disclosure is directed to a method for culling audio sources. The culling may be performed either at the core audio decoding stage or at the audio rendering stage.

[0075] "Culling" methods are aimed at removing perceptually irrelevant audio sources. For example, if a listener begins to experience the noise of heavy rain, a culling process will remove the audio source associated with the audio of birds chirping from the rendering pipeline.

[0076] In general, to detect when an audio source is not relevant, the following aspects are taken into account: distance (from the listener to the audio object), Rendering gain (of audio objects).

[0077] problem However, these existing detection techniques have various problems. For example, conventional techniques cannot detect a subclass of perceptually irrelevant audio sources. That is, the subclass includes sources that are inaudible but whose corresponding distance (gain) values ​​are higher / lower than a culling threshold. The threshold may be based on distance and gain. For example, if the source distance is greater than the distance threshold D, the source is culled. Alternatively, if the source gain value is less than the gain threshold G, the source is culled. The problem is that under existing techniques, some audio sources may ultimately be classified as "unculled," but they may be perceptually irrelevant or inaudible.

[0078] Solution The present disclosure is directed to performing culling of audio sources based on additional information, such as information about perceptual energy / loudness related aspects, which allows for improvements in estimating perceptual relevance and setting "culling" application conditions.

[0079] One aspect of the present disclosure considers relevant information during culling. The relevant information is the loudness of a single audio source. The loudness information (i.e., value) can be obtained from: 1) Metadata from existing audio standards set by the MPEG Audio Group, including MPEG-H 3D Audio (ISO_IEC_23008-3) or MPEG-D Part 4 (Dynamic Range Control). For example, metadata from legacy content may be presented. 2) Information from an encoder established by the MPEG Audio Group, such as an encoder compatible with the MPEG-H 3D Audio or MPEG-I Audio standards. For example, loudness information may be estimated and transmitted via the MPEG-H Audio Stream (MHAS) packet payload. 3) Information from the MPEG-I renderer, e.g., estimated in real time using the audio content and rendering gain.

[0080] Each of these sources can be considered individually or in combination. For example, a combination of different loudness data sources (1), (2), (3) may be used for certain types of applications (e.g., content with no or unreliable loudness information, social VR audio content).

[0081] Another aspect of the present disclosure considers information about the relationship between the loudness of a single audio source and the loudness of the entire rendered audio output. Some implementations may exclude this source, for example, in the case of an SNR-based relationship. Loudness information can be signaled and processed differently depending on different loudness definitions and measurement methods (e.g., short-term, instantaneous, EBU R128 definition).

[0082] A further aspect of the present disclosure considers information regarding the relationship between the short-term spectral energy of a single audio source and the spectral energy of the final rendered audio output relative to a psychoacoustic masking model.

[0083] FIG. 6 illustrates an example method for culling audio sources according to this disclosure.

[0084] The method includes a first step S601. In step S601, one or more values ​​and / or relationship data related to perceptual relevance information can be obtained for a plurality of audio sources. The audio sources can be received and / or predetermined. The values ​​indicate the loudness of each audio source. In one example, the relationship data can indicate a relationship between the loudness of a single audio source and the loudness of the entire rendered audio output (excluding the single source). The single audio source is one of the plurality of audio sources. In another example, the relationship data relates to a relationship between the short-term spectral energy of the single audio source and the spectral energy of the final rendered audio output related to the psychoacoustic masking model. The single audio source is one of the plurality of audio sources.

[0085] In step S602, each of the values / data from S601 is verified to see if it satisfies a corresponding predetermined condition threshold. An example is an "SNR" value at which a single audio source is considered a "useful signal" and the remaining audio sources are considered "noise." The SNR value is then compared to a threshold. The threshold may be preset or may be dynamically determined. Audio sources having values ​​that satisfy the corresponding condition threshold are considered perceptually irrelevant. A subset of perceptually irrelevant sources is then selected from the multiple sources from S601.

[0086] In step S603, audio culling is performed on perceptually irrelevant audio sources from S602 that satisfy a condition threshold. Step S603 outputs information about the state of the audio source (culled or unculled). For example, the state of an audio source, which may be "culled" or "unculled," can be represented by a Boolean variable indicating "true" or "false." For example, a Boolean variable called "isCulled" is specified and initialized to "false." After processing, it can be set to "true" by the assignment operator "isCulled=true." This variable is then passed up the processing chain to exclude this audio source for rendering. In a subsequent step (not shown), this culling information is provided to an audio renderer. The audio renderer uses the culling information in combination with the audio sources to determine which audio sources to render.

[0087] Audio Source Clustering The present disclosure is further directed to performing clustering of audio sources. The "clustering" stage involves replacing some audio sources with a smaller number of modified sources. Clustering refers to grouping based on certain criteria (e.g., proximity of audio sources, such as sources that are close to each other). Clustering may be performed either in the core audio decoding stage or in the audio rendering stage.

[0088] problem The current solution provided by the MPEG standardized audio group does not support clustering of audio sources with the goal of reducing the load on the computational and memory systems without degrading the perceptual quality of the resulting rendered audio output. However, such clustering functionality is an important feature that needs to be supported in future standardized solutions for standardized MPEG audio functionality.

[0089] Solution The present disclosure relates to clustering of audio sources based on additional aspects related to perceptual localization for estimating perceptual relatedness and setting "clustering" application conditions.

[0090] A further aspect of the present disclosure is to determine clustering based on positional proximity with respect to the listener position (i.e., close source positions). For example, if the distance between N audio source positions is less than T distance % of the distance between the listener and the closest source, these audio sources are replaced by one or more audio sources M. Here, M < N, preferably M = 1. The resulting M audio sources need to include a weighted downmix signal or a multi-channel audio signal.

[0091] T distance % represents a variable indicating a percentage value (e.g., 5%). When the listener explores a scene containing multiple audio sources (i.e., the original N audio sources), the distance between the listener and the closest source can be calculated. This distance to the closest source can be represented as the variable Tc. Further, the distances between each of the remaining audio sources within that scene can also be calculated. These individual distances are compared to Tc. If the distance between two sources is less than T distance % of Tc, these sources are clustered (grouped together). Since the comparison is made for all audio sources, a cluster can add more members (audio sources). Assuming that a total of N audio sources are clustered, these sources may be replaced by M audio sources. M is the number of audio sources to be rendered instead of N. Thus, the M audio sources still represent the same cluster.

[0092] Figure 3 shows an example of an abstract visualization of clustering based on positional proximity. As shown in Figure 3, cluster A301 contains audio sources 301A that are closest to each other in terms of distance. Similarly, cluster B302 contains different sources 302B that are closest to each other in terms of distance. Conversely, source X303 is located further away from both the audio sources 301A and 302B of clusters A301 and B302, and thus cannot be assigned to either cluster.

[0093] A further aspect of the present disclosure is to determine clustering based on directional proximity (i.e., proximity of source azimuth and elevation angles) with respect to the listener position. Note that the listener's perceptual localization ability has a finer resolution with respect to azimuth than elevation. For example, if the azimuth and elevation angles associated with the listener position of N audio sources are each less than T azimuth and T elevation degrees, these audio sources are replaced by one or more audio sources M (M < N, preferably M = 1) that include a weighted downmixed signal or a multichannel audio signal. T azimuth and T elevation are, for azimuth and elevation respectively, thresholds in degrees, e.g., 5 degrees, representing directional proximity. When a listener explores a scene containing multiple audio sources (i.e., the original N audio sources), the angular / directional difference between any two sources as seen from the listener's pose can be calculated (based on azimuth and elevation). If the angular / directional difference between two sources is less than T azimuth and T elevation these sources are clustered (grouped together). Since the comparison is made for all audio sources, clusters can add or remove more members (audio sources).

[0094] Figure 4 shows an example of an abstract visualization of clustering based on directional and / or positional / distance proximity. As seen in Figure 4, from the perspective of azimuth or elevation angle between any two sources (e.g., A2 and An) as seen from listener L, the angular / directional difference is each T azimuth degrees and T elevation degrees or less. Thereafter, these sources are clustered together. Also, since the distances between these audio sources (A, A1, A2, A3, and An) are very close to each other, it can also be assumed that they are clustered based on positional or distance proximity.

[0095] Figure 7 shows an exemplary method of clustering audio sources according to the present disclosure.

[0096] This method includes a first step S701. In step S701, for a plurality of audio sources, one or more perceptual relevance information values of each audio source can be determined. The perceptual relevance information can be one of i) positional proximity and / or ii) directional proximity.

[0097] In step S702, the values from S701 are each verified to determine whether they meet the corresponding predetermined condition threshold. Audio sources having values that meet the corresponding condition threshold are added to the cluster.

[0098] In step S703, all sources within the cluster (e.g., all N audio sources) are replaced by a smaller number of audio sources (e.g., M, where M < N) for rendering. The replacement can include a weighted downmix operation on the entire set or a subset of the N audio sources. It can also be obtained by simply pruning some of the sources.

[0099] The output of step S703 may be a set of M rendered audio sources that, when rendered, replace the original N audio sources without degrading perceptual quality.

[0100] Audio Source Typecasting This disclosure is directed to a "typecasting" stage that converts one or more audio sources from one type to another. The resulting types are expected to be more computationally efficient to render. "Typecasting" may be performed in either the core audio decoding stage or the audio rendering stage.

[0101] For example, a "typecasting" method may determine whether the distance between the listener and each of multiple audio sources is large enough (e.g., by comparing the distance between the listener and the audio source to a threshold). If the distance is large enough, these audio sources (e.g., the diffuse sound of a waterfall) can be replaced with an audio object / source (e.g., a point source of a distant waterfall). Alternatively, these audio point sources (e.g., raindrops) can be replaced with a single HOA (e.g., rain noise) signal.

[0102] problem The current solution provided by the MPEG Standardized Audio Group does not support typecasting of audio sources with the goal of reducing the load on computational and memory systems without degrading the perceptual quality of the resulting rendered audio output. However, such typecasting is an important feature that needs to be supported in future standardized solutions for standardized MPEG audio features.

[0103] Solution The present disclosure is directed to typecasting of audio sources based on one or more of the following conditions: Positional and directional proximity (see the clustering discussion above for details on measuring these parameters). Acoustic / optical occlusion / level of abstraction.

[0104] The level of acoustic / optical occlusion / abstraction can be measured by considering several aspects, such as the acoustic properties (transmission, absorption, reflection coefficients, etc.) of occluders blocking the listener's direct view and the distance of the audio source from the listener. For example, if several audio sources (multi-channel audio sources) are occluded and acoustically significant, they can be simply replaced with a single point audio source or FOA. A source is considered occluded if it is obscured by an occluder and is therefore not visible to the listener. The occlusion is described by acoustic properties such as transmission, reflection, and absorption coefficients. In this example, the occluded multi-channel audio source can be replaced with a mono audio source derived from the dominant channel of the multi-channel audio or from a weighted mix of the audio sources. Alternatively, the occluded multi-channel audio source can be replaced with a stereo audio source.

[0105] FIG. 8 illustrates an exemplary method for typecasting an audio source according to this disclosure.

[0106] In step S801, each audio source of a plurality of audio sources may be evaluated to obtain one or more values ​​for each audio source, where the values ​​indicate perceptual relevance information, for example, the values ​​may indicate (i) positional proximity, (ii) directional proximity, and / or (iii) level of acoustic / optical occlusion / abstraction.

[0107] In step S802, for each audio source, each value is evaluated to determine whether it satisfies a corresponding predetermined condition threshold. For example, the threshold may be 5 degrees. If the resulting value is less than 5 degrees, it is considered close in terms of the directional proximity criterion. The resulting source that satisfies the threshold is classified as a typecasting source.

[0108] We can then say that the obtained value satisfies the condition threshold.

[0109] In step S803, the typecasting source from step S802 is converted to another type. The audio type may be multi-channel audio (e.g., stereo, 5.0, 7.0, 11.0), mono audio, or Ambisonics (FOA, HOA).

[0110] Controlling the culling, clustering and typecasting stages The culling, clustering and typecasting methods described below can be performed alone or in various combinations.

[0111] Each of these methods can be implemented in the decoding and rendering stages of an audio decoder that is compatible with standards set by the MPEG Audio Group of the ISO / IEC organization, such as the MPEG-I Immersive Audio standard. For example, each of these stages can be implemented, alone or in combination, as part of the MPEG-I rendering process or as an external tool for the MPEG-I Immersive Audio standard.

[0112] Each method can be (de) / activated by control information. The control information can be provided as a condition. The control information can be in the form of parameters that are processed by the system and / or device. The parameters can be defined for each (virtual) object.

[0113] Applying the methods of the present invention, namely culling, clustering, and typecasting, may result in a change in the state of an audio source. For example, the state of an audio source may change from "unculled" to "culled," or vice versa. When such a change occurs, the actual change in rendering should ideally occur when the loudness of the corresponding audio source is low, to avoid sudden changes in the loud signal. In this case, the control information may be a loudness or energy threshold that sets the threshold at which such a change is triggered. Furthermore, the control information may include, for example, a maximum number of allowed changes within a certain period of time to avoid frequent and repetitive changes within a short period of time. As an example, for an audio signal, only one change within one minute is allowed. Alternatively, the control information may be set by a listener or decoder / renderer application.

[0114] Examples of control information parameters that can be used to control the application of the perceptual audio culling, clustering, and typecasting rendering stages include activation or deactivation parameters. For example, such parameters can indicate when a particular audio source is disabled (e.g., signaling to disable a narrator's voice signal). An example of activation can be applied to a particular audio source in a scene, such as an automobile audio source. Typically, an automobile sound can be active (e.g., engine starting, car running) or inactive (e.g., engine off, parked car) in a scene. In this case, activation means that the methods of the present invention (i.e., culling, clustering, typecasting) can be applied to the automobile audio source. This cannot be applied to, for example, a narrator's voice signal. In this case, deactivation control is used because it is expected to always render the voice signal.

[0115] Another example of a control information parameter is data grouping. For example, grouping data can be a set of audio source IDs that belong together as sound-emitting car parts (tires, engine, horn, etc.). These audio sources can be grouped if the directional difference between these sources relative to the listener's position is less than 5 degrees (directional proximity). Grouping data can be used to control culling / clustering / type-casting, respectively. For example, grouping data can provide a means to define a set of audio sources within a specific area (e.g., sound-emitting parts of a car) or a specific signal category (e.g., ambient sounds of rain and wind).

[0116] When grouping data is available, the encoder, decoder, and / or renderer can define culling and clustering strategies for combining or removing audio sources (e.g., objects). For example, if all audio sources in a group are below a certain threshold loudness, the group is combined into one object (loudness-based culling). Or, if the azimuth / elevation angles of all objects in a group are within a certain respective range, the group is combined into one object (orientation-based clustering). Or, if one or many objects / sources in a group are acoustically occluded, the group is combined into one object / source.

[0117] Another example of a control information parameter relates to prioritization. For example, the control information can provide a prioritization regarding the signal perceptual relevance (and its informational importance) to the listener (and the listener's focus of attention). For example, a prioritization parameter can control when culling of irrelevant audio sources is applied (instead of leveling / EQing them) for a "cocktail party effect" reproduction to improve speech intelligibility.

[0118] Another example of a control information parameter is the visibility of the graphical representation of the audio source to the listener (i.e., whether the visual representation of the audio source is visible to the user). For example, an audio source may be associated with a visual representation that (i) is not within the video-rendered viewport (behind the listener) and (ii) is obstructed by an optically non-transparent occluder. Such audio sources may be treated differently depending on their nature and application scenario. For example, thresholds for positional or directional proximity conditions may differ depending on the visibility of the virtual object. The control information parameter defines one or more sets of pre-defined condition thresholds for the object visibility. To avoid frequent "switching" between the culling, clustering and type-casting rendering stages, the following aspects can be used: hysteresis (dependence of the current decision on the history of previous decisions); Distance relative to the listener's position (and its change over time - position velocity), The direction (and its change over time - angular velocity) relative to the listener's position.

[0119] FIG. 9 illustrates an exemplary use of control information to control one or more of the culling, clustering, and / or typecasting methods of the present invention. For example, at 900, listener pause (i.e., position) information and multiple audio sources P may be received. At 901A, control information A may be received. At 901B, control information B may be received. At 901C, control information C may be received. In various embodiments, 901A, 901B, and 901C may each be implemented individually or in combination. Control information A illustrates an example where the information controls both the culling method and the clustering method. Control information B and C are used to control only the clustering method and the typecasting method, respectively.

[0120] 10 illustrates control information parameters that may define one or more sets of predetermined condition thresholds for object / source visibility. On the one hand, source A is clearly visible (not obstructed by occlusion) from the listener's position, so a set of condition thresholds X is applied to perform the audio culling, clustering, and typecasting of the present invention. On the other hand, source B is obstructed by occlusion, so another set of condition thresholds Y is applied to perform the audio culling, clustering, and typecasting of the present invention. The control information here is whether the visual representation of the audio source is visible to the user.

[0121] Apparatus for carrying out methods according to the present disclosure Finally, this disclosure also relates to apparatus (e.g., computer-implemented apparatus) for performing the methods and techniques described throughout this disclosure. FIG. 11 shows an example of such an apparatus 1100. In particular, the apparatus 1100 includes a processor 1110 and a memory 1120 coupled to the processor 1110. The memory 1120 can store instructions for the processor 1110. The processor 1110 can also receive input data 1130, particularly as appropriate depending on the use case and / or implementation. The processor 1110 can be adapted to perform the methods / techniques described throughout this disclosure and to generate corresponding output data 1140 depending on the use case and / or implementation.

[0122] interpretation A computing device implementing the above-described techniques may have the following exemplary architecture: Other architectures are possible, including architectures with more or fewer components. In some implementations, the exemplary architecture includes one or more processors (e.g., a dual-core Intel® processor), one or more output devices (e.g., an LCD), one or more network interfaces, one or more input devices (e.g., a mouse, keyboard, touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components may communicate and exchange data over one or more communication channels (e.g., a bus), which may utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0123] The term "computer-readable medium" refers to any medium that participates in providing instructions to a processor for execution, including, but not limited to, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media, including, but not limited to, coaxial cables, copper wire, and fiber optics.

[0124] The computer-readable medium may further include an operating system (e.g., a Linux operating system), a network communications module, an audio interface manager, an audio processing manager, and a live content distributor. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system performs basic tasks such as: recognizing input from and providing output to network interfaces and / or devices, tracking and managing files and directories on computer-readable media (e.g., memory and storage), controlling peripheral devices, and managing traffic on one or more communications channels. The network communications module includes various components for establishing and maintaining network connections (e.g., software for implementing communications protocols such as TCP / IP, HTTP, etc.).

[0125] The architecture may be implemented in a parallel processing or peer-to-peer infrastructure, or in a single device having one or more processors. The software may include multiple software components or may be a single body of code.

[0126] The described features may be advantageously implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, by a computer to perform a particular activity or bring about a particular result. Computer programs may be written in any type of programming language, including compiled or interpreted languages ​​(e.g., Objective-C, Java), and may be arranged in any form, such as a stand-alone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0127] Processors suitable for the execution of a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer also includes, or is operatively coupled to communicate with, one or more mass storage devices for storing data files. Such devices include magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0128] To provide for user interaction, the functionality may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a retinal display device, for displaying information to the user. The computer may have a touch surface input device (e.g., a touch screen) or a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0129] The functionality may be implemented in a computer system including back-end components such as a data server, or a computer system including middleware components such as an application server or an Internet server, or a computer system including front-end components such as a client computer having a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include, for example, a LAN, a WAN, and the computers and networks forming the Internet.

[0130] A computer system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a client device (e.g., to display data to and receive user input from a user interacting with the client device). Data generated by the client device (e.g., the result of a user interaction) can be received from the client device by the server.

[0131] One or more computer systems may be configured to perform particular operations by installing software, firmware, hardware, or a combination thereof on the system that, when in operation, causes the system to perform the operations. One or more computer programs may be configured to perform particular operations by containing instructions that, when executed by a data processing device, cause the device to perform the operations.

[0132] While one or more implementations have been described by way of example and in terms of specific embodiments, it is to be understood that the one or more implementations are not limited to the disclosed embodiments. To the contrary, the implementations are intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0133] Similarly, although operations are shown in a particular order in the figures, this should not be understood to require that such operations be performed in the particular order shown, or sequentially, and that all of the illustrated operations be performed, to achieve desirable results. In certain environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can typically be integrated together in a single software product or packaged in multiple software products.

[0134] Unless otherwise indicated, and as will be apparent from the discussion that follows, throughout this disclosure, discussions using terms such as "processing," "computing," "calculating," "determining," "analyzing," etc. refer to the operations and / or processing of a computer or computing system or similar electronic computing device that manipulates and / or converts data represented as physical, e.g., electronic, quantities into other data similarly represented as physical quantities.

[0135] References throughout this disclosure to "one embodiment," "some embodiments," or "exemplary embodiment" mean that a particular feature, structure, or characteristic described in connection with an exemplary embodiment is included in at least one exemplary embodiment of this disclosure. Thus, the appearances of the phrases "in one exemplary embodiment," "some exemplary embodiments," or "in an exemplary embodiment" in various places throughout this disclosure do not necessarily all refer to the same exemplary embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more exemplary embodiments.

[0136] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe a common object merely indicates that different instances of similar objects are being referred to and does not imply that the objects being described must be in a given sequence in time, space, ranking, or in any other way.

[0137] It is also to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As used herein, the use of "including," "comprising," or "having" and variations thereof is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled" are used broadly and encompass both direct and indirect mounting, connecting, supporting, and coupling.

[0138] In the following claims and in the description herein, any one of the terms "comprise," "have," "consist," or "consist" is broad and means including at least the elements / features that follow it, but not excluding others. Therefore, when used in a claim, the term "comprise" should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the scope of an expression "an apparatus includes A and B" should not be limited to the apparatus consisting of only elements A and B. The term "have," when used herein, is also broad and means including at least the elements / features that follow it, but not excluding others. Therefore, "have" is synonymous with "include," meaning "comprise."

[0139] It should be understood that in the foregoing description of exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in fewer than all features of a single foregoing disclosed exemplary embodiment. Thus, the claims following the description are expressly incorporated into this description, with each claim standing on its own as a separate exemplary embodiment of the present disclosure.

[0140] Furthermore, although some exemplary embodiments described herein include some features but not other features included in other exemplary embodiments, it will be understood by those skilled in the art that combinations of features from different exemplary embodiments are meant to be within the scope of the present disclosure and form different exemplary embodiments. For example, in the following claims, any of the claimed exemplary embodiments can be used in any combination.

[0141] In the description provided herein, numerous specific details have been set forth. However, it will be understood that example embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in order not to obscure an understanding of the description of the present invention.

[0142] Thus, while what is believed to be the best mode of the disclosure has been described, those skilled in the art will appreciate that other and further modifications may be made without departing from the spirit of the disclosure, and that all such changes and modifications are intended to be within the scope of the disclosure. For example, any mathematical formulas described above are merely representations of procedures that may be used. Functions may be added or deleted from block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to methods described within the scope of the disclosure.

[0143] Enumerated exemplary embodiments Various aspects and implementations of the present disclosure can be understood from the following non-claimed enumerated example embodiments (EEE).

[0144] (EEE1) A method for processing a plurality of audio sources, said method comprising: determining a respective culling value for each of the plurality of audio sources; for each of the plurality of audio sources, comparing a culling value for each of the plurality of sources to a threshold to determine whether it is a culling source; outputting information identifying a culling subset of the plurality of audio sources, the culling subset including the culling sources identified by the comparison; A method comprising:

[0145] (EEE2) The method described in EEE1, wherein the culling value is based on at least one of a loudness value, a relationship between the loudness of a single source and the overall loudness of the entire rendered audio, and a relationship between the short-term spectral energy of a single audio source and the overall spectral energy of the final rendered audio.

[0146] (EEE3) The method of EEE1, further comprising a step of rendering the plurality of audio sources, wherein the rendering step renders only audio sources that are not part of the culling subset of the plurality of audio sources.

[0147] (EEE4) A method for processing multiple audio sources, said method comprising: determining a respective clustering value for each of the plurality of audio sources; for each of the plurality of audio sources, comparing the clustering value of each of the plurality of sources to a threshold to determine whether it is a cluster source; outputting information identifying a cluster subset of the plurality of audio sources based on the comparison; A method comprising:

[0148] (EEE5) The method described in EEE4, wherein the clustering values ​​indicate locational proximity.

[0149] (EEE6) The method of EEE5, wherein the threshold is a fraction of the distance between the listener and the nearest source.

[0150] (EEE7) The method of EEE6, wherein the comparison compares the distance between the audio source and the listener to the threshold.

[0151] (EEE8) The method described in EEE4, wherein the clustering values ​​indicate directional proximity.

[0152] (EEE9) The method according to EEE8, wherein the threshold is at least one of an azimuth angle or an elevation angle between sources relative to a listener position.

[0153] (EEE10) The method of EEE4, further comprising a set of clusters, and rendering the set of clusters by an audio renderer.

[0154] (EEE11) A method for processing a plurality of audio sources, said method comprising: determining a respective typecasting value for each of the plurality of audio sources; for each of the plurality of audio sources, comparing the typecasting value of each of the plurality of sources to a threshold to determine whether it is a typecasting source; outputting information identifying a new source type that replaces the original source type; A method comprising:

[0155] (EEE12) A method of processing control information, said control information determining whether to perform a method described in one or more of EEE1, 4 and / or 11.

[0156] (EEE13) The method of EEE12, wherein the control information is one of activation or deactivation parameters, grouping data, prioritization information, and / or visibility information.

[0157] (EEE14) A method according to any one of EEE1 to 13, wherein said method is carried out in accordance with the standards set by the MPEG Audio Group of ISO / IEC.

[0158] (EEE15) The method according to EEE14, wherein said standard is the MPEG-I Immersive Audio standard.

Claims

1. 1. A method of rendering an audio source, the method comprising: determining, for each of a plurality of audio sources, a rendering control value based on one or more parameters indicative of the perceptual relevance of each audio source; for each of the plurality of audio sources, comparing the rendering control value to a perceptibility threshold to determine whether each audio source meets a perceptual irrelevance criterion; modifying an audio source among the plurality of audio sources whose rendering control value satisfies the threshold to obtain a modified audio source; rendering unmodified audio sources of the plurality of audio sources against the modified audio sources; A method comprising:

2. the one or more parameters include loudness of each audio source; The method of claim 1 , wherein the rendering control value is determined based on a loudness value.

3. 3. The method of claim 1, wherein the one or more parameters include a relationship between the loudness of a single audio source and the loudness of an audio output obtained by rendering the multiple audio sources excluding the single audio source.

4. 4. The method of claim 1, wherein the one or more parameters comprise a relationship between the short-term spectral energy of a single audio source and the spectral energy of an audio output obtained by rendering the multiple audio sources.

5. The method of claim 4 , wherein the rendered audio output is associated with a psychoacoustic masking model.

6. modifying the audio source whose rendering control value satisfies the threshold includes culling the audio source; The method according to any one of claims 1 to 5, wherein the modified audio sources correspond to a culling subset of the plurality of audio sources.

7. The method of claim 6 , wherein rendering the unmodified audio sources of the plurality of audio sources relative to the modified audio sources comprises not rendering audio sources included in the culling subset.

8. The method of any preceding claim, wherein the one or more parameters include the positional proximity of an audio source to a listener position.

9. 9. The method of claim 8, wherein the rendering control value is determined based on a distance of each audio source position of the audio source for which the rendering control value is determined relative to another audio source, and the rendering control value is compared to a portion of the distance between the listener position and a nearest source position.

10. The method of any preceding claim, wherein the one or more parameters include directional proximity to a listener position.

11. The method of claim 10 , wherein the rendering control values ​​are determined based on at least one of an azimuth angle or an elevation angle between the listener position and each audio source position for which the rendering control values ​​are determined.

12. modifying the audio sources includes clustering audio sources whose rendering control values ​​satisfy the threshold; The method according to any one of claims 8 to 11, wherein the modified audio source corresponds to a cluster audio source.

13. 13. The method of claim 12, wherein rendering unmodified audio sources of the plurality of audio sources relative to the modified audio sources comprises replacing the cluster audio sources with fewer audio sources and rendering the fewer audio sources as the modified audio sources.

14. the one or more parameters include one or more of acoustic occlusion, optical occlusion, or abstraction; The method according to any of the preceding claims, wherein the rendering control values ​​are determined based on the occlusion or abstraction level of each audio source.

15. modifying the audio source includes converting a type of audio source for which the rendering control value satisfies the threshold; The method according to any one of claims 8 to 14, wherein the modified audio source corresponds to a typecast audio source.

16. 16. The method of claim 15, wherein rendering the unmodified audio source against the modified audio source of the plurality of audio sources comprises rendering the typecast audio source as the modified audio source.

17. The method according to any one of claims 8 to 16, wherein, for an audio source of the plurality of audio sources that has a directional pattern, if the rendering control value satisfies the threshold, no directivity calculation is performed.

18. 18. The method of any of claims 1 to 17, further comprising receiving control information indicating whether to perform modification of some or all of the audio sources whose rendering control values ​​satisfy the threshold.

19. The method of claim 18 , wherein the control information is one of activation or deactivation parameters, grouping data, prioritization information, and / or visibility information.

20. 1. An apparatus for rendering an audio source, the apparatus including one or more processors configured to perform a method, the method comprising: determining, for each of a plurality of audio sources, a rendering control value based on one or more parameters indicative of the perceptual relevance of each audio source; for each of the plurality of audio sources, comparing the rendering control value to a perceptibility threshold to determine whether each audio source meets a perceptual irrelevance criterion; modifying an audio source among the plurality of audio sources whose rendering control value satisfies the threshold to obtain a modified audio source; rendering unmodified audio sources of the plurality of audio sources against the modified audio sources; Including, equipment.

21. 20. An apparatus comprising: a processor; and a memory coupled to said processor and storing instructions for said processor, said processor adapted to perform the method of any one of claims 1 to 19.

22. A program comprising instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 19.

23. A computer-readable storage medium storing the program according to claim 22.

Images