Determination of sound source direction

Abstract

The invention discloses a device and a method for determining the direction of a sound source. The apparatus includes at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least: receive metadata, the metadata includes direct sound information providing an indication of a balance between direct sound and other sounds; determining a plurality of metrics associated with a plurality of directions based on the received metadata, where the metrics depend on an amount of the direct sound; and for a respective plurality of directions, accumulating values dependent on the plurality of metrics to enable the sound source direction to be determined.

Description

[0001] This application is a divisional application of the Chinese patent application “Determination of the direction of the sound source” with application number 202011235509.6 (filing date is November 6, 2020). Technical Field

[0002] Embodiments of the present disclosure are directed to enabling determination of sound source direction. Background Art

[0003] It is desirable to determine the direction of one or more sound sources.

[0004] Current techniques may, for example, perform digital signal processing to convert delays between related audio events captured at microphones of known physical offsets into positions of the sound sources of the audio events. Summary of the invention

[0005] According to various but not all embodiments, there is provided an apparatus comprising:

[0006] means for providing a plurality of metrics associated with a plurality of directions, wherein the metrics depend on an amount of direct sound propagated for a respective plurality of directions; and

[0007] Means for accumulating, for a respective plurality of directions, values ​​dependent on a metric associated with the respective plurality of directions so as to enable determination of a direction of a sound source.

[0008] In some but not all examples, determining metrics for a plurality of frequencies respectively during the same time period, wherein, for a corresponding plurality of directions, accumulating values ​​dependent on the metrics associated with the corresponding plurality of directions to enable determination of the sound source direction comprises:

[0009] Values ​​dependent on a metric associated with the respective plurality of directions are accumulated across a plurality of time periods and a plurality of frequencies for the respective plurality of directions to enable determination of the sound source direction.

[0010] In some but not all examples, values ​​that depend on a metric associated with the corresponding multiple directions are accumulated across multiple time periods and multiple frequencies using frequency-dependent weightings for the corresponding multiple directions, in which accumulation the weightings weight values ​​that depend on the metric for a first frequency more than values ​​that depend on the metric for a second, different frequency.

[0011] In some but not all examples, the apparatus includes: means for receiving metadata, wherein the metadata includes direct sound information providing an indication of a balance between direct sound and other sounds for respective multiple directions; and means for determining a plurality of metrics using the metadata.

[0012] In some but not all examples, the metadata includes a value that depends on the ratio of the energy of the direct sound to the energy of other sounds.

[0013] In some but not all examples, the apparatus includes means for receiving metadata, wherein the metadata includes:

[0014] first metadata, which is used for the accumulation and defines corresponding multiple directions; and

[0015] The second metadata, which is used to determine the metric, depends on the amount of direct sound propagated relative to the amount of reverberant sound or ambient sound for the respective plurality of directions.

[0016] In some but not all examples, the metadata includes metadata for multiple time periods and for multiple frequencies in the same time period.

[0017] In some but not all examples, for a respective plurality of directions, accumulation depends on values ​​of metrics associated with the respective plurality of directions to enable determination of the sound source direction using a direct audio model defined by a kernel function that controls how the metrics are accumulated over an angular distribution.

[0018] In some but not all examples, the metric is a direct energy metric that indicates the energy of propagated direct sound for a corresponding plurality of directions.

[0019] In some but not all examples, the direct energy metric is or is based on a ratio of the energy of direct sound transmitted to the total sound energy transmitted.

[0020] In some but not all examples, the direct energy metric is based on the total sound energy transmitted, which is determined from the received audio signal or determined from the received metadata.

[0021] In some but not all examples, the direct energy metric is based on mapping a non-directional energy value or a total energy value to a direct energy value representing directly propagated sound energy.

[0022] In some but not all examples, the direct energy metric is determined by applying a nonlinear function to the non-directional energy values ​​or the total energy values ​​prior to mapping to provide the direct energy metric, or by applying a nonlinear function to the direct energy values ​​after mapping to provide the direct energy metric.

[0023] In some but not all examples, the mapping includes weighting the total energy value using a ratio of the energy of the direct sound transmitted to the total sound energy transmitted, or weighting the indirect energy value using a ratio of the energy of the direct sound transmitted to the energy of the indirect sound transmitted.

[0024] In some but not all examples, the mapping is frequency dependent.

[0025] In some but not all examples, the apparatus includes means for determining the direction of the sound source based on accumulating, for a respective plurality of directions, values ​​that depend on a metric associated with the respective plurality of directions.

[0026] In some but not all examples, the apparatus is configured to process audio signals recorded by the apparatus to provide a plurality of metrics associated with a plurality of directions.

[0027] In some but not all examples, the apparatus includes means for processing one or more received audio signals to provide a plurality of metrics associated with a plurality of directions, or includes means for receiving a plurality of metrics associated with a plurality of directions.

[0028] In some but not all examples, the device is configured to render audio.

[0029] In some but not all examples, the device is configured to capture audio.

[0030] According to various but not all embodiments, there is provided a method comprising:

[0031] providing a plurality of metrics associated with a plurality of directions, wherein the metrics depend on an amount of direct sound propagated for a respective plurality of directions; and

[0032] For a respective plurality of directions, values ​​dependent on a metric associated with the respective plurality of directions are accumulated to enable determination of the sound source direction.

[0033] According to various but not all embodiments, there is provided a computer program that, when run on a computer, performs the following operations:

[0034] providing a plurality of metrics associated with a plurality of directions, wherein the metrics depend on an amount of direct sound propagated for a respective plurality of directions; and

[0035] For a respective plurality of directions, values ​​dependent on a metric associated with the respective plurality of directions are accumulated to enable determination of the sound source direction.

[0036] According to various but not all embodiments, examples are provided as claimed in the following claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0038] Figure 1An example embodiment of the subject matter described herein is shown;

[0039] Figure 2A Another example embodiment of the subject matter described herein is shown;

[0040] Figure 2B Another example embodiment of the subject matter described herein is shown;

[0041] Figure 3 Another example embodiment of the subject matter described herein is shown;

[0042] Figure 4 Another example embodiment of the subject matter described herein is shown;

[0043] Figure 5 Another example embodiment of the subject matter described herein is shown;

[0044] Figure 6 Another example embodiment of the subject matter described herein is shown;

[0045] Figure 7 Another example embodiment of the subject matter described herein is shown;

[0046] Figure 8 Another example embodiment of the subject matter described herein is shown;

[0047] Fig. 9 Another example embodiment of the subject matter described herein is shown. DETAILED DESCRIPTION

[0048] Figure 1 An example of an apparatus 100 is shown, comprising means 110 for providing a plurality of metrics 10 and means 120 for accumulating values ​​depending on the metrics 10 to enable determination of a sound source direction.

[0049] Figure 2A The diagram shows the n The amount of direct sound transmitted S n In this example, direction D n It is relative to the origin O and is indicated by arrows. Direction D n This is an example, other directions D can be used n Direction D n The thickness of the arrow on D indicates the direction n The amount of direct sound S propagating toward the origin O n Direction D n It can be defined using a three-dimensional relationship (θ, φ), where θ, φ are orthogonal angles, such as the elevation angle φ (which can be expressed as the polar angle 90°-φ) and the azimuth angle θ.

[0050] Figure 2B The determined sound source direction SSD is illustrated.

[0051] Return to reference Figure 1 , the device 100 comprises:

[0052] To provide multiple directions n A component 110 of a plurality of associated metrics 10, wherein the metrics 10 are for a corresponding plurality of directions D n Depends on the amount of direct sound S transmitted n ;as well as

[0053] For multiple directions D n , the accumulation depends on the corresponding multiple directions D n The value of the metric 10 is associated with a means 120 for determining the sound source direction SSD.

[0054] Direct sound is sound that is propagated directly from a source to an origin O. It is distinguished from indirect sound (reverberant sound) that is reflected by one or more surfaces and from ambient sound that may be produced by several unrelated sound sources.

[0055] The direction of one or more sound sources is derived from the accumulated value 20. In some examples, the determination of the sound source directions SSD may occur at the device 100. However, in other examples, the device 100 may output the accumulated value 20 so that the determination of the sound source directions SSD may occur elsewhere.

[0056] The sound source direction SSD may be determined as a direction pointing to multiple sound sources at a certain time or a direction pointing to one (or more) sound sources at different times.

[0057] The determination of the cumulative value 20 (and optionally the sound source direction SSD) can be performed using a low bit rate stream, since only the metric 10 determined from the audio signal is required, and not necessarily the original captured audio signal.

[0058] Determination of the cumulative value 20 indicating the sound source direction SSD depends on the direct sound propagating from a certain direction, and in particular, on the balance between the direct sound and other sounds. Figure 2A , the direct sound propagating from the direction D1 has the best balance of direct sound relative to other sounds—the amount of direct sound is the largest in the direction D1.

[0059] For direction D1, other sounds are indirect sounds received from direction D1 (if any) and all sounds received from other directions (e.g., D2, D3, D4, D5, D6, D7, D8).

[0060] The metric 10 is agnostic (independent) to the audio capture device that captured the audio signal used to generate the metric 10. Therefore, as a result, the cumulative value 20 and the determination of the sound source direction are agnostic (independent) to the audio capture device.

[0061] For each direction D n The metric 10 may be determined separately for a plurality of frequencies in the same time period.

[0062] Assume that there is a n The associated metric X n and a time-frequency tile defined by the time period Δt and the frequency range Δf. Thus, there exists a n The value of V n . For direction D n , the cumulative value 20 is given by ∑ Δt ∑ Δf V n The cumulative value 20 is given for each direction D n The cumulative value of .

[0063] Therefore, for the corresponding multiple directions D n To accumulate depends on the corresponding multiple directions D n The means 120 for associating the values ​​of the metric 10 comprises: means for determining the values ​​of the metric 10 for the corresponding plurality of directions D n , accumulated across multiple time periods and multiple frequencies depending on the corresponding multiple directions D n The value of the metric 10 is associated to enable means 120 for determining the sound source direction SSD.

[0064] In some but not all examples, for the corresponding plurality of directions D n Depends on the amount of direct sound S transmitted n The metric 10 is a direct sound metric indicating the amount of direct sound propagated for a corresponding plurality of directions.

[0065] Figure 3 An example of the apparatus 100 as described above is illustrated. In this example, the apparatus 100 comprises:

[0066] means 110 for providing a plurality of direct sound metrics 10 associated with a plurality of directions, wherein the direct sound metrics 10 indicate an amount of direct sound propagated for a respective plurality of directions; and

[0067] Means 120 for accumulating, for a respective plurality of directions, values ​​dependent on the direct sound measure 10 associated with the respective plurality of directions to enable determination of the sound source direction.

[0068] In this example, the apparatus 100 comprises means for receiving metadata 30 and means 110 for determining a plurality of direct sound metrics 10 using the metadata 30 .

[0069] The metadata 30 includes direct sound information which provides an indication of the amount of direct sound propagated for a respective plurality of directions. The indication of the amount of direct sound propagated for a certain direction may, for example, indicate a balance between direct sound and other sounds.

[0070] The direct sound metric 10 may be, for example, a direct energy metric E* indicating the propagated direct sound energy for a corresponding plurality of directions θ. The direct energy metric E* may be, for example, based on mapping a non-directional energy value or a total energy value E to a direct energy value E* representing the directly propagated sound energy. The mapping may be defined at least in part by metadata. For example, the metadata 30 may define a factor r that is multiplied by the non-directional energy value or the total energy value E to provide the direct energy value E*.

[0071] Therefore, the metadata 30 may include a value r that depends on the ratio of direct sound energy to other sound energies. In some examples, the other sounds may be the amount of reverberant sound or the amount of ambient sound. If R1 is the ratio of direct sound energy to other sound energies, and R2 is the ratio of direct sound energy to total energy (direct sound energy plus other sound energies), then the following mathematical relationship exists between R1 and R2: R2=R1 / (R1+1); R1=R2 / (1-R2). Therefore, the value that depends on the ratio of direct sound energy to other sound energies may be R1 or R2. The value may be the ratio of direct sound energy to other sound energies, or the ratio of direct sound energy to total energy (wherein the total energy is the direct sound energy plus the other sound energies).

[0072] The metadata 30 is dynamic and changes over time. Thus, at one moment, the device 100 responds to metadata 30 including first direct sound information to enable determination of a first set of sound source directions, wherein the first direct sound information provides an indication of a first balance between direct sound and other sounds for a plurality of directions, and later, the device 100 responds to different metadata 30 including second different direct sound information to enable determination of a second set of sound source directions different from the first set of sound source directions, wherein the second different direct sound information provides an indication of a second balance between direct sound and other sounds for a plurality of directions that is different from the first balance.

[0073] In some examples, metadata 30 is device independent.

[0074] In some examples, metadata 30 is included in a data structure with different fields for different directional indexes. Each field indicates, for a direction, the amount of direct sound propagating from that direction. In some examples, the directional index is fixed and predetermined. In other examples, the direction associated with the directional index is specified in metadata 30.

[0075] In some but not all examples, the metadata 30 includes: first metadata 31 and second metadata 32. The first metadata 31 is used to controllably define the corresponding plurality of directions θ. The second metadata 32 is used to determine the metric 10. The second metadata 32 depends on the amount of direct sound propagated for the corresponding plurality of directions. For example, it includes direct sound information that provides an indication of the balance between direct sound and other sounds (e.g., a ratio of direct sound energy to other sound energy).

[0076] In some but not all examples, metadata 31, 32 includes metadata for multiple time periods and for multiple frequencies in the same time period. It defines the amount of direct sound propagated for different directions in the time-frequency domain. For example, it defines how much sound energy is directional and how much sound energy is indirect for each direction in the time-frequency domain.

[0077] The metadata 30 may be determined by analyzing the audio signal 50, such as Figure 4 As shown.

[0078] The means for performing the analysis may be at the audio capture device at or near the microphone, or at a remote downstream processing device.The analysis may be performed at the apparatus 100 , or the results of the analysis may be provided to the apparatus 100 , for example.

[0079] In this example, analysis 60 of the audio signal 50 forms metadata 30 containing spatial parameters such as direction, energy ratios, etc.

[0080] For the first metadata 31, an example parameterization is a directional parameter θ(k,n) in each frequency band, while for the second metadata 32, an example parameterization is the associated direct to total energy ratio r(k,n) in each frequency band, where k is the frequency band index and n is the time frame index.

[0081] If a mobile phone has microphones for producing audio signals, the uneven shape of the audio capture device means that one of the axes (the front-to-back axis) is very thin (<10mm), which in practice means that the differential information between the different microphone signals in the front-to-back axis is only a few millimeters. In other words, this axis has a lower signal-to-noise ratio compared to the long left-right axis, for example. Therefore, for such a device, a suitable choice is to only perform a binary front-to-back selection and estimate the delay parameters for calculating the azimuth direction using the microphones in the longer axis. The following procedure can be used to estimate the direction parameters in the horizontal plane from a device with three microphones (a pair of left and right microphones in the front and one microphone in the back):

[0082] Estimate the delay parameter d(k,n) between the left and right microphones (eg, of a camera phone in landscape mode) over a frequency band.

[0083] The delay parameter d'(k,n) is normalized between -1 and 1 so that the maximum value is obtained when the sound arrives from the axis determined by the left and right microphone pair.

[0084] An angle parameter indicating a direction between -90 and 90 degrees is obtained by α(k,n)=arcsin(d′(k,n)).

[0085] Estimate the delay parameter d between the front and rear microphones (one of which can be the same as the left and right microphones) in the frequency band fb (k, n).

[0086] Determine the delay parameter d before and after fb (k, n) is a positive or negative value. If the delay parameter indicates that the sound is coming from behind, the azimuth direction is determined by θ(k, n) = 180° - α(k, n). Otherwise, θ(k, n) = α(k, n).

[0087] Furthermore, if the device has a fourth microphone, the elevation direction can also be estimated, for example, by:

[0088] Another microphone pair (upper and lower microphone pair) with vertical displacement is selected, and another delay parameter d between them in the frequency band is estimated. e (k, n).

[0089] The delay parameter d′ e (k,n) is normalized between -1 and 1 so that the maximum value is obtained when the sound arrives from the axis determined by the upper and lower microphone pairs.

[0090] By φ(k,n)=arcsin(d′ e (k, n)) obtains the elevation direction parameter (elevation angle).

[0091] If the axis defined by the upper and lower microphone pair is offset by 90° relative to the axis defined by the left and right microphone pair, the angle may be modified to account for the offset.

[0092] The azimuth angle θ(k, n) direction and the elevation angle φ(k, n) direction can be combined to represent the direction D in three-dimensional space n The vector θ(k,n)=(θ(k,n), φ(k,n)). The direction θ(k,n) indicates the dominant arrival direction of the propagating sound.

[0093] The second metadata 32 may include a direct to total energy ratio r.

[0094] In some but not all examples, the energy ratio parameter r may be estimated based on a normalized (between -1 and 1) cross-correlation parameter c(k,n) between the microphone pair at frequency band k.

[0095] One method of determining the energy ratio parameter r is to convert the normalized cross-correlation parameter c(k,n) to the diffuse field normalized cross-correlation parameter c D (k, n) are compared and passed Determine the ratio.

[0096] The normalized cross-correlation is:

[0097]

[0098] Where S2(b,n) and S1(b,n) are microphone signals in the frequency domain, * represents the complex conjugate, Re represents the real part, b is the frequency bin, and b low (k) is the lowest interval of frequency band k, b high (k) is the highest interval of frequency band k. One of the microphone signals may be adjusted in time.

[0099] The diffuse-field normalized cross-correlation depends on the type of microphone (e.g., omnidirectional or cardioid), distance, and orientation. For example, for two omnidirectional microphones at a distance d, it can be estimated as:

[0100]

[0101] Here, f is the frequency and c is the speed of sound.

[0102] The diffuse-field normalized cross-correlation expresses the correlation between two microphones as a function of frequency in a completely diffuse (reverberant) sound field. If the sound field contains only reverberation, this is the correlation (average).

[0103] If the sound field contains only one sound source under anechoic conditions (no reverberation), the normalized cross-correlation is 1 (or close to 1).

[0104] 1-cD (k,n) represents the "range" over which the normalized cross-correlation is expected.

[0105] c(k,n)-c D (k,n) indicates how much larger the normalized cross-correlation value is compared to the diffuse field cross-correlation value.

[0106] Calculating the ratio of these two gives an estimate of the direct sound compared to all sounds.

[0107] This is just one example of how to estimate the direct to total energy ratio. There are many ways to estimate it, and the best method may depend on the microphone array type.

[0108] The metadata 30 includes a direction θ(k, n) as first metadata 31 and an energy ratio r(k, n) as second metadata 32 .

[0109] At block 110 of the apparatus 100, the energy ratio r(k,n) is used to determine a metric 10. The metric 10 indicates the relevance of the direction of arrival estimate θ(k,n) for localizing a sound source.

[0110] The metric 10 may be, for example, a direct energy metric indicating the propagated direct sound energy for a corresponding plurality of directions. The direct energy metric may be, for example, based on mapping a non-directional energy value or a total energy value E to a direct energy value E* representing the directly propagated sound energy. The mapping may be defined at least in part by the second metadata 32. For example, the second metadata 32 may define a factor r that is multiplied with the non-directional energy value or the total energy value E to provide the direct energy value E*.

[0111] The factor r may be the ratio of the direct sound energy transmitted to the total sound energy transmitted, or the ratio of the direct sound energy transmitted to the indirect sound energy transmitted.

[0112] In some examples, the factor r may be adjusted to be frequency dependent.

[0113] The overall time-frequency tile energy E=e(k,n) can be obtained as the time-frequency tile energy value e of all audio channels j j The sum of (k, n):

[0114]

[0115] In some examples, if it is known that some audio channels provide more relevant and reliable estimates, a weighted sum may be used. The time-frequency tile energy e of the more relevant channel j j (k, n) is multiplied by a larger weight factor (e.g., a weight factor greater than 1), the less correlated time-frequency tile energy e j(k, n) are multiplied by a smaller weighting factor (e.g., a weighting factor less than 1) and then added. The resulting sum can be renormalized by dividing by the sum of the weighting factors for all channels. In some examples, the weighted sum can be used to emphasize audio channels that are known to provide more relevant and reliable estimates.

[0116] refer to Figure 3 , for each directional estimate θ(k,n), at block 110, a metric 10 is calculated based on the energy e(k,n) and the ratio r(k,n).

[0117] For example, the direct energy measure (E*) can be calculated using the following formula:

[0118] E*=w(k,n)=w e (k,n)r(k,n)

[0119] Among them, the energy value w e (k,n) depends on the energy e(k,n) of the time-frequency tile.

[0120] The energy ratios r(k,n) received in the second metadata 32 are used as weighting factors to emphasize direction estimates corresponding to strong directional sounds.

[0121] In some but not all examples, the energy value w e (k,n) is the energy e(k,n) of the time-frequency tile.

[0122] In other examples, the energy value w e (k,n) is based on the energy e(k,n) of the time-frequency tile, e.g., it is functionally dependent on the energy e(k,n) of the time-frequency tile.

[0123] For example, a nonlinear function can be used to convert the energy e(k,n) of a time-frequency tile into an energy value w e (k, n).

[0124] In some but not all examples, the nonlinear function is a compression function that rescales and reduces the influence of dominant sources relative to weaker sources. In the presence of one or more strong dominant sources, this improves the detectability of weak secondary sources.

[0125] For example, a logarithmic function may be used.

[0126] w e (k, n) = 10loge(k, n)

[0127] In this example, weighting by the energy ratio r(k,n) is applied in addition to the nonlinear function to produce w′ e (k, n). In other examples, a nonlinear function may be applied after weighting with the energy ratio r(k, n), such as

[0128] w(k,n)=10log[e(k,n)r(k,n)]

[0129] In some examples, the non-linear function may be arranged to cut off energy values ​​above or below a specified threshold.

[0130] For example, a truncated non-linear function may be used.

[0131]

[0132] Among them, w′ e (k, n) is calculated as

[0133]

[0134] Here, C e is a constant that specifies the range of energy levels considered for direction estimation, for example 60 dB.

[0135] In some, but not all, examples, frequency-dependent weighting factors are used to emphasize direction estimates corresponding to frequencies of interest. For example, detection of speaker direction benefits from weighting that emphasizes speech frequency ranges.

[0136] Accumulating values ​​depending on the metric 10 associated with the respective multiple directions across multiple time periods and multiple frequencies may use frequency-dependent weightings for the respective multiple directions, wherein the weightings weight the values ​​depending on the metric 10 for a first frequency more than the values ​​depending on the metric 10 for a second, different frequency in the accumulation. The first frequency is within the frequency range of human speech and the second frequency is outside the frequency range of human speech.

[0137] In the example described above, a frequency-dependent weighting factor w may be applied before applying the nonlinear function k (k) For example:

[0138] w(k,n)=w e (k, n)*r(k, n)*w k (k)

[0139] The device 100 includes a device for accumulating, for corresponding multiple directions, the values ​​of the values ​​depending on the corresponding multiple directions D n The value w(k,n) of the associated metric 10 enables means 120 to determine the sound source direction SSD. The provided metric 10 may be, for example: r(k,n); w e (k, n) and r(k, n); w e (k, n), r(k, n), and w k (k); or w(k, n).

[0140] A plurality of metrics 10 are associated with a plurality of directions and depend on the amount of direct sound S propagated for the respective plurality of directions. n .

[0141] As previously described, the audio signal energy E may be determined in the time-frequency domain and combined with the received direct and total energy ratio r to produce a metric. For example, the energy ratio r(k,n) is used as a weighting factor to emphasize directional estimates corresponding to strong directional sounds.

[0142] In some examples, the time-frequency domain energy value e(k, n) or e j (k, n) is included in the metadata 30 as third metadata.

[0143] In other examples, the time-frequency domain energy value (e.g., E or e(k, n) or e j (k, n)) are not included in the metadata 30, and they are estimated by the processing component 70 processing the received audio signal 50, such as Figure 5 As shown.

[0144] The audio signal 50 may be processed by a processing component 70 to compute a (time-frequency) energy distribution (such as a spectrogram).

[0145] The energy of the time-frequency tile e(k,n) can be calculated based on the audio signal s j (t) 50. The audio signal 50 may, for example, represent one or more audio channels j over time.

[0146] Time domain signal j (t) is transformed into a time-frequency domain signal S j (b, n), where b is the frequency band index. The transform may be performed using any suitable transform, for example a short time Fourier transform (STFT) or a complex modulated quadrature mirror filter bank (QMF).

[0147] For each channel, the energy e(k,n) is calculated in the frequency band:

[0148]

[0149] Among them, B low (k) is the lowest frequency band k, B high (k) is the highest frequency band k.

[0150] Return to reference Figure 3, the audio signal 50 and its associated metadata are provided to the apparatus 100. The apparatus uses the at least one audio signal 50 and the associated metadata 32 to determine a plurality of metrics 10 associated with a corresponding plurality of directions. The at least one audio signal 50 may be used to determine an energy value E from the audio signal 50 using the processing component 70, and the metadata 32 may be used to convert the energy value E into a direct energy value E*. For example, the time-frequency energy distribution (spectrogram) and energy value (as well as the direct energy value) of the audio signal may be determined for each frequency band.

[0151] The audio signal 50 may be a microphone signal (eg, a signal captured from one or more microphones and / or from a microphone array) or a transmitted audio signal.

[0152] For example, a sound scene may be captured using high quality microphones that capture high quality audio signals, and a microphone array microphone may be used for directional (and other spatial metadata) analysis.

[0153] In some but not all examples, the metadata 30 and the audio signal 50 are provided to the device 10 as metadata-assisted spatial audio (MASA). The metadata-assisted spatial audio reduces the multi-channel audio signal to a combination of a smaller number of transmitted audio channels and metadata. During rendering, the combination of the smaller number of transmitted audio channels and the metadata can be used to recreate the perceived effect of the original multi-channel audio signal.

[0154] A MASA signal may be created from a microphone array signal (eg, from a mobile device), and the transmitted audio signal and metadata may be used to recreate the perceived effect of the originally captured sound scene.

[0155] A MASA signal may be created from a multi-channel loudspeaker signal (eg, 5.1).

[0156] In some but not all examples, the audio signals 50 represent left and right audio channels respectively, ie, there is an audio signal for the left audio channel and an audio signal for the right audio channel.

[0157] refer to Figure 3 The device 100 includes a device for targeting multiple directions D n Cumulative depends on multiple directions D n The value of the metric 10 is associated to enable the sound source direction SSD component 120 to be determined.

[0158] The apparatus 100 comprises means 120 for mapping directional information (eg, first metadata 31) to a corresponding metric 10 representing directly received energy.

[0159] The temporal distribution model of the directions of arrival is accumulated using the metrics 10 and their associated directions, eg provided by the first metadata 31. The statistical model of the direction estimation may be implemented as a histogram of the direction estimates for a given time interval weighted with the corresponding metrics 10.

[0160] The direction histogram indicates the most likely direction of the sound source in the time interval.The sound source direction can be estimated, for example, based on the maximum value of the histogram, or by fitting a specific function to the histogram data.

[0161] The histogram may include an angular distribution of bins. By binning the values ​​depending on the metric 10 (and hence the second metadata 22) according to the first metadata 21, accumulation occurs over the angular distribution of the bins based on the associated directions. For example, the "direct" energy E* is binned by direction of arrival. Accumulation will integrate the direct energy E* into the direction bins in time and frequency.

[0162] In one example, a histogram M(n) contains a set of bins m l (n), each interval corresponds to a specific direction. Each metric 10 is assigned to an associated interval. The cumulative value of each interval represents an approximation of the probability of receiving sound from that direction.

[0163] Histogram bins can be evenly or unevenly distributed. Higher density histogram bins can be used in directions that are more important than other directions, or in areas where the angular resolution of the direction estimate is higher.

[0164] The histogram can be a rectangular projection based on azimuth and elevation to a plane so that the histogram intervals are evenly distributed on the azimuth axis and the elevation axis (or polar axis). Alternatively, the histogram intervals can be evenly distributed on the surface of a sphere. For example, a regular polyhedron (such as an octahedron or a dodecahedron) can be used as the basis of a geometric shape. Then, each face of the polyhedron is iteratively subdivided into smaller triangles with vertices on the surface of the sphere until sufficient resolution is reached. The resulting vertices are used as the direction of the histogram intervals. Other geometric shapes can be used.

[0165] In some examples, values ​​w(k, n′) that depend on a metric 10 associated with the corresponding multiple directions are accumulated for corresponding multiple directions to enable determination of the sound source direction using a direct audio model defined by a kernel function K that controls how the values ​​w(k, n′) that depend on the metric 10 are accumulated over an angular distribution.

[0166] For example, the direction estimate θ(k, n) increases the histogram interval m of the histogram M(n) corresponding to time n l The value in (n):

[0167]

[0168] where K(l, θ(k, n′)) is a kernel function that specifies how the effect of the direction estimate θ(k, n′) is geometrically distributed across the direction D l The values ​​w(k,n′) depending on the metric 10 are accumulated over all frequency bands k and N consecutive time frames.

[0169] The kernel function K(l, θ(k, n)) can be implemented in different ways.

[0170] In one implementation, the index l of the histogram bin closest to the estimated direction θ(k,n) is located, and the weight w(k,n) is added to the corresponding histogram bin:

[0171]

[0172] In another implementation, a plurality of histogram bins near the estimated direction θ(k, n′) are updated using a kernel function that typically has a peak at the estimated direction and decreases as the angular difference increases. The kernel propagates a metric. The kernel can be, for example, a Gaussian window or based on a Hanning window. In one implementation, the histogram bin closest to the direction estimate is found, and a pre-computed weight is used for the nearest neighboring bin. In another implementation, a weight can be calculated for each nearby bin based on the angular difference between the direction estimate and the direction corresponding to the bin.

[0173] The time interval N for accumulating the direction estimate can be a time frame. A longer time interval can be used to reduce the random variation of the histogram data. However, increasing the time window length also reduces the accuracy of the direction estimate of the mobile source.

[0174] In order to make a continuous estimate of the direction of change, a histogram can be generated for consecutive moments. Histogram data from previous moments is useful in reducing random changes, but old measurements eventually need to be discarded to track moving sound sources. A simple implementation method is to apply a forgetting factor to the previous histogram. The histogram M(n) used to detect the source direction is obtained as follows: M(n) = αM′(n) + (1-α)M(n-1), where α determines how quickly previous information is forgotten by the system, and M′(n) is the histogram accumulated for a moment n.

[0175] exist Figure 3 In the example, the sound source direction is determined by the component 130 for determining the sound source direction SSD based on the cumulative value.

[0176] For example, the estimate of the sound source direction θ ss(i,n) is based on a histogram, where i is the index of the detected sound source. The means 130 for estimating the direction of the sound source may be at the apparatus 100 or elsewhere.

[0177] The component 130 for estimating the direction of a sound source may be able to estimate the direction of a variable number of sound sources, and in some cases, may not be able to estimate the direction of any sound source (e.g., if all sound sources are silent). However, the component 130 for estimating the direction of a sound source is generally capable of estimating the direction of one or several sound sources.

[0178] The direction corresponding to the global maximum of the histogram can be used as the direction of the dominant sound source θ ss (n) instructions.

[0179] Multiple sound sources can be located by detecting several local maxima in the histogram.

[0180] Local maxima that are close to each other may originate from a single sound source, and in some examples may be considered to be a single sound source.

[0181] The following iterative algorithm can be used to detect the direction of multiple sound sources:

[0182] (i) Set M(1,n)=M(n) to initialize the histogram for detecting the direction of the first source.

[0183] (ii) Find the index l of the maximum value in the current histogram M(i, n). Obtain an estimate of the sound source direction θ ss (i, n), as the direction corresponding to the maximum value.

[0184] (iii) If the expected number of source directions are found, stop the iteration. Otherwise, continue.

[0185] (iv) Remove the contribution of source i from the histogram to obtain a new histogram M(i+1, n). This can be done by subtracting a kernel from the histogram. The kernel is centered at bin index l and its value in that bin is m. l , and the kernel value decreases with increasing angular distance from the interval. In other words, the interval at the maximum value i found has zero value, and its immediate neighboring intervals are smoothly attenuated in the resulting histogram M(i+1,n). The width and shape of the kernel can correspond to typical sound source peaks in the histogram.

[0186] (v) Go to step (ii) to find the next sound source i+1.

[0187] A spatial smoothing filter can also be applied to the histogram before detecting the source direction.

[0188] The accuracy of the estimated sound source direction can be improved by interpolation. For example, the interpolation of the peak direction can be based on a small number of histogram bins that are geometrically close to the bin with the maximum value. For example, the weighted sum of the directions of the neighboring bins of the bin with the maximum value can be calculated as follows:

[0189]

[0190] where N(l) is a set of indices of intervals that are geometrically close to the maximum interval l (including the interval l itself). For example, N(l) can be a set of geometric intervals that are immediately adjacent to the interval l and l itself. p represents the direction of the interval p. In practice, the direction can be represented by a point on a unit sphere in a 3D Cartesian coordinate system, and the azimuth and elevation of the interpolated direction are calculated based on the weighted sum of the point coordinates.

[0191] The kernel function K can be adjusted according to the characteristics of the acoustic environment (such as reverberation or background noise distribution) to increase accuracy in situations where more or less active spatial filtering is needed to average out the noise in the estimated signal to improve the reliability of sound source detection. Different applications may also require more or less sensitive directional estimates, for example to react faster to changes or to apply long-term averaging to identify obvious low-level sound sources.

[0192] like Figure 6 As shown in the figure, the sound source direction SSD (e.g. θ ss (n)) may be further processed at block 200. Block 200 may or may not be part of apparatus 100.

[0193] In one example, the direction θ of the sound source ss (n), spatial metadata (direction θ(k,n) and energy ratio r(k,n)), and microphone array signal s j (t) is forwarded to block 200 for audio rendering.

[0194] The audio rendering block 200 synthesizes the spatial audio into an estimated direction θ with the audio focus pointing to the sound source. ss As a result, the audio focus automatically follows the detected sound source (such as a speaker).

[0195] Audio focus may be achieved, for example, by temporally adjusting the microphone signals based on the microphone arrangement and the desired focus direction and summing the temporally adjusted microphone signals. More generally, the microphone signals may be processed in the time-frequency domain with complex gains and combined to obtain a focused audio signal.

[0196] The audio focus direction can be controlled to point to a potential sound source of interest. This is useful, for example, when a user of a mobile device is shooting a video in a noisy environment and wants to capture the voice of the speaker. The method can be used to automatically detect and follow the direction of the speaker to manipulate the audio focus direction. The audio focus can amplify the sound of the source of interest and reduce noise. As a result, speech clarity can be improved, making the audio of a video shot in a noisy environment better.

[0197] Users can decide whether to focus on a certain source after audio is captured. Users can manually select the focus direction, or the focus direction can automatically follow the sound source.

[0198] In some examples, at block 200, the direction θ of the sound source ss (n) can be used as input to other sound source tracking algorithms. Such algorithms can be used for:

[0199] Mark each source with a unique identifier (tagging);

[0200] Tracking one or more mobile sources;

[0201] Smoothing out the movement of the estimated sound source direction;

[0202] Lock on one or more detected sources, even if they are not the strongest peaks in the histogram;

[0203] Ignore one or more sources based on some specific conditions.

[0204] In some examples, block 200 implements a voice-controlled user interface and automatic speech recognition.Block 200 automatically detects the direction of the speaker.

[0205] In some examples, the audio focus can be adjusted toward the speaker to reduce interfering sounds from other sound sources in the captured audio signal and improve speech recognition performance. In some examples, the direction of the speaker can be a parameter for determining an input voice command.

[0206] In some examples, block 200 is used to detect the direction of an active speaker in a conference recording for visualization of the conference and / or speaker-based selective playback of the conference.

[0207] It will be appreciated from the foregoing that the apparatus 100 may receive the metric 10 or generate the metric 10. In some examples, the apparatus 100 includes means for processing one or more received audio signals 50 to provide a plurality of metrics 10 associated with a plurality of directions. In some examples, the apparatus 100 includes means for processing received metadata (r, E) to provide a plurality of metrics 10 associated with a plurality of directions. In other examples, the apparatus 100 includes means for receiving a plurality of metrics 10 associated with a plurality of directions.

[0208] It will be appreciated from the foregoing description that there may be a pipeline that includes: (i) capturing audio, (ii) generating metadata, (iii) processing metrics based at least on the metadata 30 to produce a cumulative value for estimating the direction of one or more audio sources, (iv) determining the direction of one or more audio sources, and (v) rendering audio (and / or other processing).

[0209] The apparatus 100 comprises means for performing (iii).

[0210] In some examples, apparatus 100 includes components for performing (ii) and (iii).

[0211] In some examples, the apparatus 100 includes components for performing (ii), (iii), and (iv). In some examples, the apparatus 100 includes components for performing (ii), (iii), (iv), and (v). In some examples, the apparatus 100 includes components for performing (i), (ii), and (iii). In some examples, the apparatus 100 includes components for performing (i), (ii), (iii), and (iv). In some examples, the apparatus 100 includes components for performing (i), (ii), (iii), and (iv). In some examples, the apparatus 100 includes components for performing (iii) and (iv). In some examples, the apparatus 100 includes components for performing (iii), (iv), and (v).

[0212] Audio can be rendered using headphones or a multi-channel speaker setup.

[0213] If the metadata stream 30 contains audio energy information E (in the time-frequency domain), the audio signal 50 need not be an input to the apparatus 100. Instead, the time-frequency energy distribution is derived from the metadata 30.

[0214] In a lightweight implementation, the cumulative value based on the metric can be determined without using the time-frequency energy information. Only the time-frequency direction information 31 and the corresponding time-frequency information 32 about the relative amount of directional and ambient sound energy are used. This corresponds to e j (k, n) is set to a constant, that is, for all k and n, w e (k, n) = 1. Therefore, w(k, n) = r(k, n)w k (k) or w(k,n)=r(k,n).

[0215] In these examples, the metric 10 is a direct energy metric which is or is based on the ratio of the direct sound energy transmitted to the total sound energy transmitted. This ratio is provided in the metadata 30.

[0216] In some but not all examples, the direct energy metric is based on the total sound energy transmitted, which is determined from the received audio signal 50 or determined from the received metadata 30.

[0217] In the foregoing example, metadata 30 may be metadata for perceptually accurate spatial sound reproduction.

[0218] The metadata 30 may be generated, for example, by analyzing perceptually relevant parameters of the sound field in frequency bands and short time intervals, such as the direction of arrival of propagating sounds at the recording location.

[0219] Perceptually spatial sound can be replicated by rendering with metadata that creates a spatial perception similar to that which would occur to a listener when listening to the original sound field. As a result, the listener can perceive multiple sources, their directions and distances, and the characteristics of the surrounding physical space, among other spatial sound characteristics, just as if the listener were at the location of the capture device.

[0220] Using the parametric spatial audio method, the captured spatial audio can also be efficiently stored and / or transmitted. Instead of storing all captured microphone signals, a small number of audio channels (e.g., one or two) can be stored together with a parametric metadata representation of the spatial characteristics of the audio.

[0221] The method reduces the amount of data to be stored or transmitted for spatial audio reproduction. It also makes storage, transmission and rendering independent of the capture device.

[0222] An example of such a parameter format is the recently proposed Metadata Assisted Spatial Audio (MASA) format for the Immersive Voice and Audio Services (IVAS) codec.

[0223] Figure 7 An example of a method 300 is illustrated, the method 300 comprising:

[0224] At block 302, a plurality of metrics associated with a plurality of directions are provided, wherein the metrics depend on an amount of direct sound propagated for the respective plurality of directions; and

[0225] At block 304 , values ​​dependent on a metric associated with the respective plurality of directions are accumulated for the respective plurality of directions to enable determination of a sound source direction.

[0226] The method 300 may be modified as described in the preceding paragraphs describing the apparatus 100 .

[0227] Figure 8An example of a controller 400 is illustrated, which may be used by the device 100 or to perform other functions described. The controller 400 may be implemented as a controller circuit system. The controller 400 may be implemented only in hardware, have certain aspects only in software including firmware, or may be a combination of hardware and software (including firmware).

[0228] like Figure 8 As illustrated, the controller 400 may be implemented using instructions that implement the hardware functionality, for example, by using executable instructions of a computer program 406 in a general or special purpose processor 402 , which may be stored on a computer readable storage medium (disk, memory, etc.) to be executed by such processor 402 .

[0229] The processor 402 is configured to read from and write to the memory 404. The processor 402 may also include an output interface via which data and / or commands are output by the processor 402 and an input interface via which data and / or commands are input to the processor 402.

[0230] The memory 404 stores a computer program 406 comprising computer program instructions (computer program code) that control the operation of the apparatus 100 when loaded into the processor 402. The computer program instructions of the computer program 406 provide the means for enabling the apparatus to perform Figures 1 to 7 By reading the memory 404 , the processor 402 is able to load and execute the computer program 406 .

[0231] Thus, the apparatus 100 may include:

[0232] at least one processor 402; and

[0233] at least one memory 404 comprising computer program code,

[0234] The at least one memory 404 and the computer program code are configured to, together with the at least one processor 402, cause the apparatus 100 to at least perform:

[0235] providing a plurality of metrics associated with a plurality of directions, wherein the metrics depend on an amount of direct sound propagated for a respective plurality of directions;

[0236] For a respective plurality of directions, values ​​dependent on a metric associated with the respective plurality of directions are accumulated to enable determination of the sound source direction.

[0237] like Fig. 9As illustrated, the computer program 406 may arrive at the apparatus 100 via any suitable delivery mechanism 410. The delivery mechanism 410 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD) or a solid-state memory, an article of manufacture including or tangibly embodying the computer program 406. The delivery mechanism may be a signal configured to reliably transmit the computer program 406. The apparatus 100 may propagate or transmit the computer program 406 as a computer data signal.

[0238] Computer program instructions for causing an apparatus to at least perform the following operations or for at least performing the following operations:

[0239] providing a plurality of metrics associated with a plurality of directions, wherein the metrics depend on an amount of direct sound propagated for a respective plurality of directions;

[0240] For a respective plurality of directions, values ​​dependent on a metric associated with the respective plurality of directions are accumulated to enable determination of the sound source direction.

[0241] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some but not all examples, the computer program instructions may be distributed over more than one computer program.

[0242] Although memory 404 is illustrated as a single component / circuit, it may be implemented as one or more separate components / circuit(s), some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cached storage.

[0243] Although processor 402 is illustrated as a single component / circuit, it may be implemented as one or more separate components / circuit, some or all of which may be integrated / removable.Processor 402 may be a single-core or multi-core processor.

[0244] References to "computer-readable storage medium," "computer program product," "tangibly embodied computer program," etc. or "controller," "computer," "processor," etc. should be understood to include not only computers having different architectures (such as single / multi-processor architectures and sequential (von Neumann) / parallel architectures), but also special-purpose circuits (such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other processing circuit systems). References to "computer programs," "instructions," "code," etc. should be understood to cover software for programmable processors or firmware, such as the programmable content of hardware devices, whether instructions for a processor or configuration settings for a fixed-function device, gate array, or programmable logic device, etc.

[0245] In this application, the term "circuitry" may refer to one or more or all of the following:

[0246] (a) pure hardware circuit implementation (such as implementation using only analog and / or digital circuits); and

[0247] (b) a combination of hardware circuitry and software such as (where applicable):

[0248] (i) a combination of analog and / or digital hardware circuitry and software / firmware, and

[0249] (ii) a hardware processor (including a digital signal processor) with software, software, and any parts of a processor that work together to enable a device such as a mobile phone or server to perform various functions; and

[0250] (c) Hardware circuits and / or processors, such as a microprocessor or portion of a microprocessor, that require software (e.g., firmware) for operation, but the software may not be present if it is not required for operation.

[0251] This definition of circuitry applies to all uses of the term in this application, including any claims. As another example, as used in this application, the term "circuitry" also covers an implementation of only a hardware circuit or processor and its accompanying software and / or firmware. For example, and if applicable to a particular claim element, the term "circuitry" also covers a baseband integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0252] exist Figures 1 to 7 The blocks illustrated in the figure may represent steps in the method and / or code portions in the computer program 406. The illustration of a particular order of blocks does not necessarily mean that these blocks have a required or preferred order, but the order and arrangement of these blocks may be changed. In addition, some blocks may be omitted.

[0253] Where structural features have been described, they may be replaced by components that perform one or more of the functions of the structural features, whether that function or those functions are explicitly described or implicitly described.

[0254] The examples described above are suitable for use as enabling components for:

[0255] Automotive systems; telecommunications systems; electronic systems, including consumer electronics; distributed computing systems; media systems for generating or rendering media content, including audio, visual and audiovisual content, and mixed, mediated, virtual and / or augmented reality; personal systems, including personal health systems or personal fitness systems; navigation systems; user interfaces, also known as human-machine interfaces; networks, including cellular, non-cellular and optical networks; ad hoc networks; the Internet; the Internet of Things; virtual networks; and related software and services.

[0256] The word "comprising" as used herein has an inclusive rather than exclusive meaning. That is, X comprises Y means that X may include only one Y, or may include more than one Y. If the exclusive meaning of "comprising" is intended, it is indicated in the context by referring to "including only one..." or using "consisting of..."

[0257] In this description, reference has been made to various examples. The description of the features or functions of an example indicates that those features or functions are present in the example. Whether explicitly stated or not, the use of the term "example" or "for example" or "may" or "can" in this article indicates that such features or functions are present at least in the example described, whether or not they are described as "examples", and they may but not necessarily be present in some or all of the other examples. Therefore, "example", "for example", "may" or "can" refers to a specific instance in a class of examples. The characteristics of an instance can only be the attributes of the instance, or the attributes of the class, or the attributes of a subclass of the class, which includes some but not all instances in the class. Therefore, it is implicitly disclosed that the features described with reference to one example rather than with reference to another example can be used as part of a working combination in the other example where possible, but not necessarily in the other example.

[0258] Although the embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

[0259] Features described in the preceding description may be used in combinations other than the combinations explicitly described above.

[0260] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

[0261] Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.

[0262] The terms "a", "an", or "the" as used herein have an inclusive rather than exclusive meaning. That is, X includes one / the Y means that X may include only one Y or may include more than one Y, unless the context clearly indicates otherwise. If the exclusive meaning of "a" or "the" is intended, it will be clearly indicated in the context. In some cases, "at least one" or "one or more" may be used to emphasize the inclusive meaning, but the absence of these terms should not be regarded as inferring an exclusive meaning.

[0263] The presence of a feature (or combination of features) in a claim is a reference to the feature or (combination of features) itself, and is also a reference to a feature (equivalent feature) that achieves substantially the same technical effect. For example, equivalent features include features that are variants and achieve substantially the same result in substantially the same way. For example, equivalent features include features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0264] In the present description, reference has been made to various examples using adjectives or adjective phrases to describe features of the examples. Such feature descriptions of the examples indicate that the feature is exactly the same as described in some examples and substantially the same as described in other examples.

[0265] While an effort has been made to draw attention in the foregoing description to those features considered to be important, it should be understood that the applicant may seek protection by way of the claims for any patentable feature or combination of features heretofore referenced and / or shown in the accompanying drawings, whether or not emphasis has been placed thereon.

Claims

1. A device comprising: at least one processor; as well as at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: determining one or more directions of arrival of at least one audio signal; Identify at least one of the following: one or more energy values ​​of the at least one audio signal, or one or more ratios for the at least one or more audio signals; Accumulating, for at least one direction, a respective value associated with a first direction of arrival of the determined one or more directions of arrival based at least in part on at least one of: an energy value of the determined one or more energy values ​​associated with the first direction of arrival, or a ratio of the one or more ratios determined, wherein the at least one direction includes a plurality of directions; and Based at least in part on the accumulated corresponding values, a sound source direction of at least one sound source is determined.

2. The device according to claim 1, wherein: Determining the one or more directions of arrival of the at least one audio signal comprises the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to: For multiple time periods and multiple frequency bands, corresponding arrival directions are determined.

3. The device according to claim 1, wherein: determining the one or more energy values ​​comprises at least one memory and the computer program code is configured to, with the at least one processor, cause the apparatus to: An energy value of a sound originating from the first direction of arrival is determined, wherein the at least one audio signal comprises the sound originating from the first direction of arrival.

4. The device according to claim 3, wherein: The energy value of the sound originating from the first arrival direction includes energy values ​​for a first time period and a first frequency band of the at least one audio signal, wherein the at least one audio signal includes sounds for a plurality of time periods and a plurality of frequency bands.

5. The device according to claim 1, wherein: The determined one or more ratios are configured to indicate a ratio between a direct sound energy of the at least one audio signal and a total sound energy of the at least one audio signal.

6. The device according to claim 5, wherein: A ratio of the one or more ratios is associated with a first time period and a first frequency band of the at least one audio signal, wherein the at least one audio signal includes sounds for a plurality of time periods and a plurality of frequency bands.

7. The device according to claim 1, wherein: The sound source direction of the at least one sound source is determined based at least in part on a direction of the at least one direction associated with a highest cumulative value.

8. The device according to claim 1, wherein: The corresponding value is further accumulated based at least in part on at least one weight, wherein the at least one weight is configured to control the value accumulated into the corresponding value and the value accumulated into one or more other accumulated values, wherein the at least one weight is based at least in part on the relationship between the at least one direction and the first arrival direction.

9. The device according to claim 1, wherein: Based on determining that the first direction of arrival is closest to at least one of the at least one directions, the at least one direction is associated with the first direction of arrival.

10. The device according to claim 1, wherein: Determining the direction of the sound source comprises at least one memory and the computer program code is configured to, together with the at least one processor, cause the apparatus to: Interpolation is performed on corresponding values ​​accumulated for a direction among the plurality of directions that is closest to a direction among the plurality of directions having a maximum accumulated value.

11. A method comprising: determining one or more directions of arrival of at least one audio signal; Identify at least one of the following: one or more energy values ​​of the at least one audio signal, or one or more ratios for the at least one or more audio signals; Accumulating, for at least one direction, a respective value associated with a first direction of arrival of the determined one or more directions of arrival based at least in part on at least one of: an energy value of the determined one or more energy values ​​associated with the first direction of arrival, or a ratio of the one or more ratios determined, wherein the at least one direction includes a plurality of directions; and Based at least in part on the accumulated corresponding values, a sound source direction of at least one sound source is determined.

12. The method according to claim 11, wherein: Determining the one or more directions of arrival of the at least one audio signal comprises: For multiple time periods and multiple frequency bands, corresponding arrival directions are determined.

13. The method according to claim 11, wherein: Determining the one or more energy values ​​comprises: An energy value of a sound originating from the first direction of arrival is determined, wherein the at least one audio signal comprises the sound originating from the first direction of arrival.

14. The method according to claim 13, wherein: The energy value of the sound originating from the first arrival direction includes energy values ​​for a first time period and a first frequency band of the at least one audio signal, wherein the at least one audio signal includes sounds for a plurality of time periods and a plurality of frequency bands.

15. The method according to claim 11, wherein: The determined one or more ratios are configured to indicate a ratio between a direct sound energy of the at least one audio signal and a total sound energy of the at least one audio signal.

16. The method according to claim 15, wherein: A ratio of the one or more ratios is associated with a first time period and a first frequency band of the at least one audio signal, wherein the at least one audio signal includes sounds for a plurality of time periods and a plurality of frequency bands.

17. The method according to claim 11, wherein: The sound source direction of the at least one sound source is determined based at least in part on a direction of the at least one direction associated with a highest cumulative value.

18. The method according to claim 11, wherein: The corresponding value is further accumulated based at least in part on at least one weight, wherein the at least one weight is configured to control the value accumulated into the corresponding value and the value accumulated into one or more other accumulated values, wherein the at least one weight is based at least in part on the relationship between the at least one direction and the first arrival direction.

19. A non-transitory computer readable medium comprising program instructions stored thereon, the program instructions, when executed by at least one processor, causing the at least one processor to: determining one or more directions of arrival of at least one audio signal; Identify at least one of the following: one or more energy values ​​of the at least one audio signal, or one or more ratios for the at least one or more audio signals; Accumulating, for at least one direction, a respective value associated with a first direction of arrival of the determined one or more directions of arrival based at least in part on at least one of: an energy value of the determined one or more energy values ​​associated with the first direction of arrival, or a ratio of the one or more ratios determined, in, The at least one direction includes a plurality of directions; and Based at least in part on the accumulated corresponding values, a sound source direction of at least one sound source is determined.

20. The non-transitory computer readable medium of claim 19, wherein: The sound source direction of the at least one sound source is determined based at least in part on a direction of the at least one direction associated with a highest cumulative value.

21. The non-transitory computer readable medium of claim 19, wherein: The corresponding value is further accumulated based at least in part on at least one weight, wherein the at least one weight is configured to control the value accumulated into the corresponding value and the value accumulated into one or more other accumulated values, wherein the at least one weight is based at least in part on the relationship between the at least one direction and the first arrival direction.

Images