Directional estimation enhancement for parameterized spatial audio capture using broadband estimation

Abstract

An apparatus comprising means configured to: obtain at least one audio signal; determining at least one direction parameter of at least one propagating sound associated with the at least one audio signal; determining at least one direction of at least one sound source associated with the at least one audio signal; modifying at least one direction parameter according to the determined at least one direction; and outputting the modified at least one direction parameter.

Description

[0001] This application is a divisional application of the Chinese invention patent application entitled “Directional Estimation Enhancement for Parametric Spatial Audio Capture Using Wideband Estimation” (application number 202080064936.4, application date September 2, 2020). Technical Field

[0002] The present application relates to apparatus and methods for directional estimation enhancement of parametric spatial audio capture using wideband estimation. Background Art

[0003] It is known to capture or record spatial sound using microphone arrays such as in consumer mobile devices (such as Nokia 8) and commercial recording devices (such as Nokia OZO). Such spatial sound can be reproduced for use with headphones or multi-channel speaker settings and provide a rich audio experience. The audio signal captured by the device can be reproduced in a suitable output format within the same device or at another device. For example, after transmission to a suitable playback or receiver device as audio channels and spatial metadata or as an Ambisonic signal.

[0004] For transmission, the audio signal or channel may be compressed, for example, using Advanced Audio Coding (AAC) or MPEG-H 3D Audio compression or other suitable compression mechanism. The spatial metadata may also be compressed and sent either in the same data packet as the audio data or as a separate compressed metadata stream. In the case where the audio signal or channel and associated metadata are compressed for transmission, they are decoded prior to reproduction.

[0005] Mobile devices that may include microphone arrays may utilize parametric spatial audio capture and rendering methods to enable perceptually accurate spatial sound reproduction. Parametric spatial audio capture refers to an adaptive DSP-driven audio capture method. Specifically, the parametric spatial audio method can generally be summarized as the following operations:

[0006] 1) analysis of perceptually relevant parameters in frequency bands and short time intervals (often referred to as time-frequency slots), such as the direction of arrival of propagating sound at the recording location, and

[0007] 2) Based on the estimated spatial parameters, the spatial sound is reproduced in a perceptual sense on the rendering side.

[0008] The reproduction can be used, for example, for headphones or a multi-channel speaker setup. By estimating and reproducing the perception-relevant spatial properties (parameters) of the sound field, a spatial perception similar to that which would occur if the listener were listening to the original sound field can be produced. As a result, the listener can perceive a large number of sound sources, their directions and distances, and the properties of the surrounding physical space, as well as other spatial sound characteristics, as if the listener were at the location of the capture device.

[0009] Binaural spatial audio reproduction methods known as spatial audio capture (SPAC) processes (such as described in PCT published patents WO2018 / 091776, WO2018 / 060549, US20130044884, WO2017 / 005978, EP3320699) capture (e.g., from Nokia 8 or Nokia OZO) microphone array signals. These processes estimate the direction of arrival (DOA) and the relative energies of direct and ambient components expressed as a direct-to-total energy ratio from the microphone signals in a frequency band and synthesize, for example, a binaural signal for headphone listening, or a multi-channel loudspeaker signal for loudspeaker listening, or an Ambisonic output for YouTube 360 ​​and similar services.

[0010] Although SPAC generally produces good audio quality, the spatial analysis accuracy may not be optimal for various reasons. Potential reasons may be, for example, irregularities in the situation of the capture device, e.g., the mobile device is held in the hand in different ways or covered with a protective case, wind noise, the necessity to use AAC encoding on the audio signal before spatial analysis (which may produce worse quality analysis), and the acoustic environment in the sound field itself. However, AAC encoding is only an example of a lossy encoding that may cause problems with spatial analysis. Any other lossy codec may also cause these problems (e.g., mp3, Ogg Vorbis, etc.) Summary of the invention

[0011] According to a first aspect, a device is provided, comprising components configured to perform the following operations: obtain at least one audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determine at least one direction of at least one sound source associated with the at least one audio signal; modify at least one directional parameter based on the determined at least one direction; and output the modified at least one directional parameter.

[0012] The component configured to obtain at least one audio signal can be configured to obtain at least one of the following: at least two microphone audio signals; at least one Ambisonic audio signal; at least one spatial audio signal, wherein the at least one spatial audio signal is associated with at least one metadata parameter, wherein the at least one metadata parameter includes at least one of the following: direction or arrival direction; energy ratio; diffuseness ratio.

[0013] A component configured to determine at least one directional parameter of at least one propagated sound may be configured to determine at least one of: at least one directional parameter of at least one propagated sound; at least one directional parameter of at least one audio source; at least one directional parameter of at least one reflection from at least one audio source; and at least one energy ratio parameter associated with at least one directional parameter.

[0014] The means configured to determine at least one directional parameter of at least one propagated sound may be configured to determine at least one directional parameter associated with at least one frequency band of at least one audio signal.

[0015] At least one propagated sound associated with at least one audio signal may include at least one of: direct sound associated with at least one audio source within the space; reflected sound associated with at least one audio source within the space; sound associated with at least one virtual source; reverberant sound; ambient sound; and any superposition of direct sound, reflected sound, reverberant sound and ambient sound.

[0016] The at least one propagated sound associated with the at least one audio signal may be related to a recording location from which the at least one audio signal was captured.

[0017] A component configured to determine at least one direction of at least one sound source associated with at least one audio signal can be configured to: obtain at least one image; analyze the at least one image to determine at least one object of interest within the image; and determine at least one direction of the at least one sound source based on a position of the at least one object of interest within the at least one image and image capture parameters.

[0018] The component configured to analyze at least one image to determine at least one object of interest within the image may be configured to identify at least one speaker based on identifying a face from the at least one image and further based on identifying movement of lips within the face between images.

[0019] A component configured to determine at least one direction of at least one sound source can be configured to perform at least one of the following: analyzing at least one signal from at least one infrared sensor to determine at least one direction of at least one sound source; analyzing at least one signal from at least one sensor to determine at least one direction of at least one sound source; obtaining at least one user input signal identifying at least one direction of at least one sound source; obtaining at least one positioning signal associated with an object, and determining at least one direction of at least one sound source from the at least one positioning signal; when at least one audio signal includes two or more microphone audio signals, analyzing the at least one audio signal; and obtaining metadata associated with the at least one audio signal, the metadata including directional or positional information, wherein, when at least one directional parameter of at least one propagated sound is determined from the analysis of the at least one audio signal, at least one direction of the at least one sound source is determined from the metadata.

[0020] A component configured to modify at least one directional parameter based on at least one determined direction can be configured to: determine an angular difference between at least one directional parameter and at least one direction of at least one sound source; compare the angular difference with a threshold, wherein if the angular difference is greater than the threshold, the at least one directional parameter is passed without modification, and if the angular difference is less than the threshold, the at least one directional parameter is modified toward at least one direction of the at least one sound source.

[0021] A component configured to modify a directional parameter in at least one direction toward at least one sound source may be configured to modify the directional parameter to be in at least one direction of at least one sound source when the angular difference is less than a determined smaller threshold, and to apply an interpolation modification to the directional parameter when the angular difference is greater than the determined smaller threshold but less than the above threshold.

[0022] The component may be further configured to: determine at least one transmission audio signal based on the at least one audio signal; and output the at least one transmission audio signal having a modified direction parameter.

[0023] The component may be further configured to: when the at least one audio signal comprises at least two microphone audio signals, analyze the at least one audio signal to determine at least one metadata parameter, wherein the at least one metadata parameter comprises at least one of: an energy ratio; and a diffusion ratio; and output at least one metadata parameter with a modified directional parameter.

[0024] According to a second aspect, a device is provided, comprising components configured to perform the following operations: obtain at least one transmission audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one transmission audio signal; determine at least one direction of at least one sound source associated with the at least one transmission audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0025] A component configured to obtain at least one transmitted audio signal may be configured to receive at least one of: at least two microphone audio signals; at least one Ambisonic audio signal; at least one spatial audio signal, wherein the at least one spatial audio signal is associated with at least one metadata parameter, wherein the at least one metadata parameter comprises at least one of: direction or direction of arrival; energy ratio; diffusion ratio.

[0026] A component configured to determine at least one directional parameter of at least one propagated sound can be configured to determine at least one of: at least one directional parameter of at least one propagated sound; at least one directional parameter of at least one audio source; at least one directional parameter of at least one reflection from at least one audio source; and at least one energy ratio parameter associated with at least one directional parameter.

[0027] The means configured to determine at least one directional parameter of at least one propagated sound may be configured to determine at least one directional parameter associated with at least one frequency band of at least one transmission audio signal.

[0028] At least one propagated sound associated with at least one transmitted audio signal may include at least one of: direct sound associated with at least one audio source within the space; reflected sound associated with at least one audio source within the space; sound associated with at least one virtual source; reverberant sound; ambient sound; and any superposition of direct sound, reflected sound, reverberant sound and ambient sound.

[0029] The at least one propagated sound associated with the at least one transmission audio signal from which the at least one transmission audio signal was generated may be related to a recording location from which the at least one audio signal was captured.

[0030] A component configured to determine at least one direction of at least one sound source associated with at least one transmitted audio signal can be configured to: obtain at least one image; analyze the at least one image to determine at least one object of interest within the image; and determine at least one direction of the at least one sound source based on a position of the at least one object of interest within the at least one image and image capture parameters.

[0031] The component configured to analyze at least one image to determine at least one object of interest within the image may be configured to identify at least one speaker based on identifying a face from the at least one image and further based on identifying movement of lips within the face between images.

[0032] A component configured to determine at least one direction of at least one sound source can be configured to perform at least one of the following: analyzing at least one signal from at least one infrared sensor to determine at least one direction of at least one sound source; analyzing at least one signal from at least one sensor to determine at least one direction of at least one sound source; obtaining at least one user input signal identifying at least one direction of at least one sound source; obtaining at least one positioning signal associated with an object, and determining at least one direction of at least one sound source from the at least one positioning signal; when at least one transmitted audio signal includes two or more microphone audio signals, analyzing the at least one transmitted audio signal; and obtaining metadata associated with at least one audio signal, the metadata including directional or positional information, wherein, when at least one directional parameter of at least one propagated sound is determined from the analysis of the at least one transmitted audio signal, at least one direction of at least one sound source is determined from the metadata.

[0033] A component configured to modify at least one directional parameter based on at least one determined direction can be configured to: determine an angular difference between at least one directional parameter and at least one direction of at least one sound source; compare the angular difference with a threshold, wherein if the angular difference is greater than the threshold, the at least one directional parameter is passed without modification, and if the angular difference is less than the threshold, the at least one directional parameter is modified toward at least one direction of the at least one sound source.

[0034] A component configured to modify a directional parameter in at least one direction toward at least one sound source may be configured to modify the directional parameter to be in at least one direction of at least one sound source when the angular difference is less than a determined smaller threshold, and to apply an interpolation modification to the directional parameter when the angular difference is greater than the determined smaller threshold but less than the above threshold.

[0035] According to a third aspect, a method is provided, which comprises: obtaining at least one audio signal; determining at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determining at least one direction of at least one sound source associated with the at least one audio signal; modifying at least one directional parameter based on the determined at least one direction; and outputting at least one modified directional parameter.

[0036] Obtaining at least one audio signal may include obtaining at least one of: at least two microphone audio signals; at least one Ambisonic audio signal; at least one spatial audio signal, wherein the at least one spatial audio signal is associated with at least one metadata parameter, wherein the at least one metadata parameter includes at least one of: direction or direction of arrival; energy ratio; diffusion ratio.

[0037] Determining at least one directional parameter of at least one propagated sound may include determining at least one of: at least one directional parameter of at least one propagated sound; at least one directional parameter of at least one audio source; at least one directional parameter of at least one reflection from at least one audio source; and at least one energy ratio parameter associated with at least one directional parameter.

[0038] Determining at least one directional parameter of at least one propagated sound may include determining at least one directional parameter associated with at least one frequency band of at least one audio signal.

[0039] At least one propagated sound associated with at least one audio signal may include at least one of: direct sound associated with at least one audio source within the space; reflected sound associated with at least one audio source within the space; sound associated with at least one virtual source; reverberant sound; ambient sound; and any superposition of direct sound, reflected sound, reverberant sound and ambient sound.

[0040] The at least one propagated sound associated with the at least one audio signal may be related to a recording location from which the at least one audio signal was captured.

[0041] Determining at least one direction of at least one sound source associated with at least one audio signal may include: obtaining at least one image; analyzing the at least one image to determine at least one object of interest within the image; and determining at least one direction of the at least one sound source based on a position of the at least one object of interest within the at least one image and image capture parameters.

[0042] Analyzing the at least one image to determine at least one object of interest within the image may include identifying at least one speaker based on identifying a face from the at least one image and further based on identifying movement of lips within the face between images.

[0043] Determining at least one direction of at least one sound source may include performing at least one of the following: analyzing at least one signal from at least one infrared sensor to determine at least one direction of at least one sound source; analyzing at least one signal from at least one sensor to determine at least one direction of at least one sound source; obtaining at least one user input signal identifying at least one direction of at least one sound source; obtaining at least one positioning signal associated with an object, and determining at least one direction of at least one sound source from the at least one positioning signal; when at least one audio signal includes two or more microphone audio signals, analyzing the at least one audio signal; and obtaining metadata associated with the at least one audio signal, the metadata including directional or positional information, wherein, when at least one directional parameter of at least one propagated sound is determined from the analysis of the at least one audio signal, at least one direction of the at least one sound source is determined from the metadata.

[0044] Modifying at least one directional parameter based on at least one determined direction may include: determining an angular difference between at least one directional parameter and at least one direction of at least one sound source; comparing the angular difference with a threshold, wherein if the angular difference is greater than the threshold, the at least one directional parameter is passed without modification, and if the angular difference is less than the threshold, the at least one directional parameter is modified toward at least one direction of the at least one sound source.

[0045] Modifying the direction parameter in at least one direction toward at least one sound source may include: when the angular difference is less than a determined smaller threshold, modifying the direction parameter to be in at least one direction of at least one sound source, and when the angular difference is greater than the determined smaller threshold but less than the above threshold, applying an interpolation modification to the direction parameter.

[0046] The method may further include: determining at least one transmission audio signal based on the at least one audio signal; and outputting the at least one transmission audio signal having a modified directional parameter.

[0047] The method may further include: when the at least one audio signal includes at least two microphone audio signals, analyzing the at least one audio signal to determine at least one metadata parameter, wherein the at least one metadata parameter includes at least one of: an energy ratio; and a diffusion ratio; and outputting at least one metadata parameter with a modified directional parameter.

[0048] According to a fourth aspect, a method is provided, comprising: obtaining at least one transmission audio signal; determining at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; determining at least one direction of at least one sound source associated with at least one transmission audio signal; modifying at least one directional parameter based on the determined at least one direction; and outputting at least one modified directional parameter.

[0049] Obtaining at least one transmitted audio signal may include receiving at least one of: at least two microphone audio signals; at least one Ambisonic audio signal; at least one spatial audio signal, wherein the at least one spatial audio signal is associated with at least one metadata parameter, wherein the at least one metadata parameter includes at least one of: direction or direction of arrival; energy ratio; diffusion ratio.

[0050] Determining at least one directional parameter of at least one propagated sound may include determining at least one of: at least one directional parameter of at least one propagated sound; at least one directional parameter of at least one audio source; at least one directional parameter of at least one reflection from at least one audio source; and at least one energy ratio parameter associated with at least one directional parameter.

[0051] Determining at least one directional parameter of at least one propagated sound may include determining at least one directional parameter associated with at least one frequency band of at least one transmitted audio signal.

[0052] At least one propagated sound associated with at least one transmitted audio signal may include at least one of: direct sound associated with at least one audio source within the space; reflected sound associated with at least one audio source within the space; sound associated with at least one virtual source; reverberant sound; ambient sound; and any superposition of direct sound, reflected sound, reverberant sound and ambient sound.

[0053] The at least one propagated sound associated with the at least one transmission audio signal from which the at least one transmission audio signal was generated may be related to a recording location from which the at least one audio signal was captured.

[0054] Determining at least one direction of at least one sound source associated with at least one transmitted audio signal may include: obtaining at least one image; analyzing the at least one image to determine at least one object of interest within the image; and determining at least one direction of the at least one sound source based on a position of the at least one object of interest within the at least one image and image capture parameters.

[0055] Analyzing the at least one image to determine at least one object of interest within the image may include identifying at least one speaker based on identifying a face from the at least one image and further based on identifying movement of lips within the face between images.

[0056] Determining at least one direction of at least one sound source may include performing at least one of the following: analyzing at least one signal from at least one infrared sensor to determine at least one direction of at least one sound source; analyzing at least one signal from at least one sensor to determine at least one direction of at least one sound source; obtaining at least one user input signal identifying at least one direction of at least one sound source; obtaining at least one positioning signal associated with an object, and determining at least one direction of at least one sound source from the at least one positioning signal; when at least one transmitted audio signal includes two or more microphone audio signals, analyzing the at least one transmitted audio signal; and obtaining metadata associated with at least one audio signal, the metadata including directional or positional information, wherein, when at least one directional parameter of at least one propagated sound is determined from the analysis of the at least one transmitted audio signal, at least one direction of at least one sound source is determined from the metadata.

[0057] Modifying at least one directional parameter based on at least one determined direction may include: determining an angular difference between at least one directional parameter and at least one direction of at least one sound source; comparing the angular difference with a threshold, wherein if the angular difference is greater than the threshold, the at least one directional parameter is passed without modification, and if the angular difference is less than the threshold, the at least one directional parameter is modified toward at least one direction of the at least one sound source.

[0058] Modifying the direction parameter in at least one direction toward at least one sound source may include: when the angular difference is less than a determined smaller threshold, modifying the direction parameter to be in at least one direction of at least one sound source, and when the angular difference is greater than the determined smaller threshold but less than the above threshold, applying an interpolation modification to the direction parameter.

[0059] According to a fifth aspect, a device is provided, comprising at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to at least: obtain at least one audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determine at least one direction of at least one sound source associated with the at least one audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0060] The device enabled to obtain at least one audio signal may be enabled to obtain at least one of: at least two microphone audio signals; at least one Ambisonic audio signal; at least one spatial audio signal, wherein the at least one spatial audio signal is associated with at least one metadata parameter, wherein the at least one metadata parameter comprises at least one of: direction or direction of arrival; energy ratio; diffusion ratio.

[0061] The device enabled to determine at least one directional parameter of at least one propagated sound may be enabled to determine at least one of: at least one directional parameter of at least one propagated sound; at least one directional parameter of at least one audio source; at least one directional parameter of at least one reflection from at least one audio source; and at least one energy ratio parameter associated with at least one directional parameter.

[0062] The means caused to determine at least one directional parameter of at least one propagated sound may be caused to determine at least one directional parameter associated with at least one frequency band of at least one audio signal.

[0063] At least one propagated sound associated with at least one audio signal may include at least one of: direct sound associated with at least one audio source within the space; reflected sound associated with at least one audio source within the space; sound associated with at least one virtual source; reverberant sound; ambient sound; and any superposition of direct sound, reflected sound, reverberant sound and ambient sound.

[0064] The at least one propagated sound associated with the at least one audio signal may be related to a recording location from which the at least one audio signal was captured.

[0065] The device configured to determine at least one direction of at least one sound source associated with at least one audio signal may be configured to: obtain at least one image; analyze the at least one image to determine at least one object of interest within the image; and determine at least one direction of the at least one sound source based on a position of the at least one object of interest within the at least one image and image capture parameters.

[0066] The apparatus caused to analyze at least one image to determine at least one object of interest within the image may be caused to: identify at least one speaker based on identifying a face from the at least one image and further based on identifying movement of lips within the face between images.

[0067] The device configured to determine at least one direction of at least one sound source may be configured to perform at least one of the following: analyzing at least one signal from at least one infrared sensor to determine at least one direction of at least one sound source; analyzing at least one signal from at least one sensor to determine at least one direction of at least one sound source; obtaining at least one user input signal identifying at least one direction of at least one sound source; obtaining at least one positioning signal associated with an object and determining at least one direction of at least one sound source from the at least one positioning signal; analyzing at least one audio signal when the at least one audio signal includes two or more microphone audio signals; and obtaining metadata associated with the at least one audio signal, the metadata including directional or positional information, wherein, when at least one directional parameter of at least one propagated sound is determined from the analysis of the at least one audio signal, at least one direction of the at least one sound source is determined from the metadata.

[0068] The device, which is caused to modify at least one directional parameter based on at least one determined direction, can be caused to: determine the angular difference between at least one directional parameter and at least one direction of at least one sound source; compare the angular difference with a threshold, wherein if the angular difference is greater than the threshold, the at least one directional parameter is passed without modification, and if the angular difference is less than the threshold, the at least one directional parameter is modified toward at least one direction of at least one sound source.

[0069] The device, which is caused to modify the directional parameter in at least one direction towards at least one sound source, can be caused to: when the angular difference is less than a determined smaller threshold, modify the directional parameter to be in at least one direction of at least one sound source, and when the angular difference is greater than the determined smaller threshold but less than the above threshold, apply interpolation modification to the directional parameter.

[0070] The apparatus may further be caused to: determine at least one transmission audio signal based on the at least one audio signal; and output the at least one transmission audio signal having a modified direction parameter.

[0071] The device may be further caused to: when the at least one audio signal comprises at least two microphone audio signals, analyze the at least one audio signal to determine at least one metadata parameter, wherein the at least one metadata parameter comprises at least one of: an energy ratio; and a diffusion ratio; and output at least one metadata parameter with a modified directional parameter.

[0072] According to a sixth aspect, a device is provided, comprising at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to at least: obtain at least one transmission audio signal; determine at least one direction parameter of at least one propagated sound associated with the at least one transmission audio signal; determine at least one direction of at least one sound source associated with the at least one transmission audio signal; modify at least one direction parameter based on the determined at least one direction; and output at least one modified direction parameter.

[0073] According to a seventh aspect, a device is provided, comprising: an acquisition circuit configured to obtain at least one audio signal; a determination circuit configured to determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; a determination circuit configured to determine at least one direction of at least one sound source associated with the at least one audio signal; a modification circuit configured to modify at least one directional parameter based on the determined at least one direction; and an output circuit configured to output at least one modified directional parameter.

[0074] According to an eighth aspect, a device is provided, comprising: an acquisition circuit configured to obtain at least one transmission audio signal; a determination circuit configured to determine at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; a determination circuit configured to determine at least one direction of at least one sound source associated with at least one transmission audio signal; a modification circuit configured to modify at least one directional parameter based on the determined at least one direction; and an output circuit configured to output at least one modified directional parameter.

[0075] According to the ninth aspect, a computer program comprising instructions [or a computer-readable medium comprising program instructions] is provided, wherein the instructions or program instructions are used to cause a device to perform at least the following operations: obtain at least one audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determine at least one direction of at least one sound source associated with the at least one audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0076] According to the tenth aspect, a computer program [or a computer-readable medium including program instructions] is provided, wherein the instructions or program instructions are used to cause a device to perform at least the following operations: obtain at least one transmission audio signal; determine at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; determine at least one direction of at least one sound source associated with at least one transmission audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0077] According to the eleventh aspect, a non-transitory computer-readable medium comprising program instructions is provided, wherein the program instructions are used to cause a device to perform at least the following operations: obtain at least one audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determine at least one direction of at least one sound source associated with the at least one audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0078] According to the twelfth aspect, a non-transitory computer-readable medium comprising program instructions is provided, wherein the program instructions are used to cause a device to perform at least the following operations: obtain at least one transmission audio signal; determine at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; determine at least one direction of at least one sound source associated with at least one transmission audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0079] According to the thirteenth aspect, a device is provided, comprising: a component for obtaining at least one audio signal; a component for determining at least one directional parameter of at least one propagated sound associated with the at least one audio signal; a component for determining at least one direction of at least one sound source associated with the at least one audio signal; a component for modifying at least one directional parameter based on the determined at least one direction; and a component for outputting the modified at least one directional parameter.

[0080] According to the fourteenth aspect, a device is provided, comprising: a component for obtaining at least one transmission audio signal; a component for determining at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; a component for determining at least one direction of at least one sound source associated with at least one transmission audio signal; a component for modifying at least one directional parameter based on the determined at least one direction; and a component for outputting at least one modified directional parameter.

[0081] According to the fifteenth aspect, a computer-readable medium comprising program instructions is provided, wherein the program instructions are used to cause an apparatus to perform at least the following operations: obtain at least one audio signal; determine at least one directional parameter of at least one propagated sound associated with the at least one audio signal; determine at least one direction of at least one sound source associated with the at least one audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0082] According to the sixteenth aspect, a computer-readable medium comprising program instructions is provided, wherein the program instructions are used to cause an apparatus to perform at least the following operations: obtain at least one transmission audio signal; determine at least one directional parameter of at least one propagated sound associated with at least one transmission audio signal; determine at least one direction of at least one sound source associated with at least one transmission audio signal; modify at least one directional parameter based on the determined at least one direction; and output at least one modified directional parameter.

[0083] An apparatus comprises means for performing the actions of the method as described above.

[0084] A device is configured to perform the actions of the method described above.

[0085] A computer program includes program instructions for causing a computer to execute the method as described above.

[0086] A computer program product stored on a medium may cause an apparatus to perform the method described herein.

[0087] An electronic device may include an apparatus as described herein.

[0088] A chipset may include the apparatus as described herein.

[0089] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0091] Figure 1 schematically illustrates an example system suitable for implementing some embodiments;

[0092] Figure 2 According to some embodiments, Figure 1 A flowchart of the operation of the example system shown in;

[0093] Figure 3 Schematically illustrating how Figure 1 The example analysis processor shown in ;

[0094] Figure 4 According to some embodiments, Figure 3 A flowchart of the operation of an example analysis processor shown in;

[0095] Figure 5 Schematically illustrating how Figure 3 The example spatial audio signal analyzer shown in ;

[0096] Figure 6 According to some embodiments, Figure 5 Flowchart of the operation of the example spatial audio signal analyzer shown in;

[0097] Figure 7 Schematically illustrating how Figure 5 The example shown in the direction is biased towards the processor;

[0098] Figure 8 According to some embodiments, Figure 7 A flowchart of the operation of an example direction bias processor is shown in;

[0099] Fig. 9 Schematically illustrating how Figure 1 Another example analysis processor shown in;

[0100] Fig.10 Schematically illustrating how Figure 1 The example synthesis processor shown in ;

[0101] Fig.11 According to some embodiments, Fig.10 A flowchart of the operation of an example synthesis processor shown in;

[0102] Fig.12 An example device suitable for implementing the illustrated means is shown. DETAILED DESCRIPTION

[0103] Suitable apparatus and possible mechanisms for providing efficient capture and rendering of spatial audio signals are described in more detail below.

[0104] The concepts discussed in the embodiments below are based on using estimates of possible source locations and adjusting the orientation estimate closer to these possible source locations.

[0105] Furthermore, although the concept and embodiments detailing the concept are described in the context of SPAC as a standalone method, it may be used in any context where a parameterized direction is estimated from an audio signal and enables obtaining another, preferably more stable, direction estimate.

[0106] For example, the concept can be applied to the Directional Audio Coding (DirAC) method. In other words, the concept is applicable to audio codecs that apply a direction-based parameterized model, and therefore, it can be applied to, for example, the 3GPP Immersive Voice and Audio Service (IVAS) or Moving Picture Experts Group-Immersive (MPEG-I) codec.

[0107] Embodiments attempt to improve the spatial audio capture methods of the prior art, which are based on estimating at least one directional parameter in a frequency band and rendering spatial sound in the frequency band according to these directional parameters. This process is performed in short time intervals, usually referred to as audio frames, to take into account the time-varying nature of most audio signals. The frequency band of the signal of the audio frame can be referred to as a time-frequency tile. The directional parameter usually refers to the main arrival direction of the propagating sound in the recording position in the time-frequency tile. It should be noted that the direction does not necessarily point to any actual sound source, but it is a superposition of sounds from, for example, actual sound sources, reflections of sounds from hard surfaces such as walls, reverberation (which refers to diffuse late reverberation, in which the listener can no longer perceive any obvious direction of the reflection), and ambient sounds (sounds without any obvious arrival direction, such as noise).

[0108] Other parameters that may be determined include a direct to total ratio parameter in a frequency band (or a similar parameter such as a diffuseness parameter or a directional stability parameter), which generally indicates how much sound energy is coming from the indicated direction of the frequency band, and how much sound energy is ambient or surround. There may be one or more simultaneous directional estimates in each frequency band, and accordingly there may be one or more of the ratio or energy parameters indicating how directional the sound is in the frequency band.

[0109] Ideally, with such a system, when a source is active in a certain direction relative to the capture device, a corresponding amount of acoustic energy is reproduced in that direction relative to the reproduction device, thereby creating a realistic perception of the source in that direction.

[0110] There are a number of factors that may affect the quality of directional metadata at least in certain frequency bands, such as:

[0111] - Capture devices are often held in the hand (especially in the case of mobile devices). Capture devices can be held in various ways, affecting the acoustic properties of the device. High-frequency analysis can be especially erroneous due to acoustic effects associated with the hand (masking of certain frequencies, other wave phenomena like sound waves reflected / diffracted from the hand).

[0112] - Some devices may store microphone audio signals in coded form, for example, as AAC coded audio signals. These AAC audio signals are decoded, for example, in a computer where the spatial analysis is performed. AAC encoding may lead to errors in the spatial estimation (due to masked quantization noise).

[0113] - The device may be protected by a windscreen or a protective case, which may affect the acoustic properties.

[0114] - Wind or other noise sources can affect the orientation estimate.

[0115] -The acoustic environment itself often causes the directional estimate to fluctuate unnecessarily widely.

[0116] Additionally, at the stage of performing spatial analysis in frequency bands with the microphone signals, there may be various further practical reasons that influence the microphone audio signals.

[0117] Embodiments as described herein attempt to produce estimated directional metadata that is less erroneous or noisy. For example, in some embodiments the direction estimate produced is more stable and therefore fluctuates less in a random manner (noise), or is more correct and has fewer errors or drifts in a systematic manner (e.g., due to acoustic effects such as a user's hand).

[0118] Therefore, embodiments as described below attempt to reduce this effect, i.e., whenever a single prominent source is active, for example, in the case of reproducing the voice of a speaker in an acoustic space, the source is reproduced so that some frequencies can be reproduced in the appropriate direction, while some (or all) frequencies may have some directional offset, error or noise. Therefore, it can reduce the effect that a point source is not reproduced only in that direction, but rather is reproduced over an arc or area of ​​directions generally in the appropriate direction. In other words, embodiments attempt to reduce this spreading (except for intentionally performed spreading, which is known to be used to spatially synthesize the spatial reproduction effect of an extended (or extended range) sound source (e.g., a crowd, a coast)).

[0119] Thus, in embodiments, the apparatus and method aim to reproduce audio sources in a spatially accurate manner, which, in particular for speech signals, may have the effect of improving their intelligibility.

[0120] Conventional methods for determining directional parameters in a frequency band are known, such as Directional Audio Coding (DirAC) operating on a First Order Ambisonics (FOA) signal or any variant thereof.

[0121] The FOA signal can be generated by designing and applying a suitable filter (or complex gain in a frequency band) to the microphone array signal. Such gains in a frequency band can be generated, for example, for each frequency band below the spatial aliasing frequency, by measuring the impulse response of the device from (approximately) uniformly distributed points and using a least squares optimization method to derive a set of mixing gains to obtain the frequency band FOA signal from the frequency band microphone array signal.

[0122] The FOA signal has an omnidirectional component w(k,n) and three orthogonal dipole components x(k,n), y(k,n) and z(k,n), where k is the frequency index and n is the time index. Following the concept of DirAC, the direction can be determined based on the vector I(k,n):

[0123]

[0124] where Re denotes the real part and * denotes the complex conjugate. The direction of arrival is then the direction of I(k,n). Note that I(k,n) may be averaged over several frequencies and / or time indices (k,n).

[0125] DirAC can also estimate diffusivity, which is a ratio factor between 0 and 1 that determines how much of the total acoustic energy is non-directional. These methods are well documented in the literature.

[0126] There are other methods known that can operate on Ambisonics signals. For example, Harmonic Plane Wave Expansion (Harpex) and its variants estimate two simultaneous directions of arrival from the FOA signal. Another example is High Order DirAC (HO-DirAC), which estimates more than one direction of arrival in a sector from a second-order or higher-order Ambisonic signal. Such Ambisonic signals require more microphones from the device, for example, at least 9 microphones in an ideal case for second order.

[0127] A method is described in US20130230187A1 that can estimate the direction of arrival, especially at high frequencies. In this method, a vector is determined to represent each microphone in the array, and the length of the vector is the estimated energy (or amplitude) in the frequency band of the microphone signal. The direction of arrival is then the direction of the sum of these vectors.

[0128] Another method is proposed in PCT published patents WO2018 / 091776, WO2018 / 060549, and US20130044884, which is particularly relevant to mobile phone implementations. In these methods, a delay search algorithm is applied to determine the delay that produces maximum coherence between microphone signals. For example, the delay parameters between the left and right microphone signals of a mobile phone in landscape mode are determined. The delay is normalized so that the maximum delay (the sound arriving from the axis of the microphone) is -1 and 1. The angular parameter is then acos(d), where d is the estimated and normalized delay parameter. This delay and angle parameter is ambiguous: for example, if the sound comes from 30 degrees (in front) and 150 degrees (in the back), the same delay can be observed.

[0129] To resolve this ambiguity, the mobile capture method also determines a binary front-to-back selection to determine whether the sound is coming from the front or the back. This is obtained by performing a delay analysis at the front-to-back microphone pair. Due to the small microphone spacing (flat device), the delay analysis can be very "noisy", however, it can be robust enough to simply determine whether the sound is generally coming from the front or the back of the device.

[0130] The aforementioned method is particularly suitable for flat devices, where one axis is used only for front-to-back analysis. However, due to uneven performance in certain directions, this method is not optimal for non-flat devices such as VR cameras. VR cameras can use, for example, DirAC, but DirAC and similar methods only work below the spatial aliasing frequency. US20130230187A1 may be applicable above the spatial aliasing frequency, but requires significant acoustic masking. In PCT application number PCT / FI2019 / 050239, a delay search method for non-flat devices without a lot of acoustic masking in certain axes is discussed. The method forms the delays between the signals of several microphone pairs and uses vector algebra to determine the directional parameters.

[0131] The concept involves parametric spatial audio capture using microphone arrays, where a novel directional analysis solution is proposed that provides improved spatial stability with respect to the main sound source (e.g., talker).

[0132] In some embodiments, the novel directional analysis is based on combining the time-frequency domain arrival direction estimation of the propagating sound with the broadband direction estimation of the sound source. In some embodiments, the method can be summarized by the following operations:

[0133] obtaining a spatial audio signal;

[0134] Obtaining an arrival direction estimation of the propagating sound of the spatial audio signal in the time-frequency domain;

[0135] Obtain a broadband directional estimate of the sound source;

[0136] biasing the time-frequency domain directional estimation towards wideband directional estimation; and

[0137] Output the biased time-frequency domain direction estimate as direction metadata.

[0138] The arrival direction estimation of the propagating sound refers to the estimation of the main direction of arrival of the sound in the recording position for the time-frequency patch. It can be related to the actual sound source, reflection, reverberation, ambient sound, or the superposition of all these (usually the case).

[0139] A broadband direction estimate of a sound source refers to an estimate of the actual direction of a sound source (e.g., a speaker). There may be estimates for multiple sound sources. It should be noted that in some (or even any) frequency bands, the sound does not primarily come from these directions, but these directions are the directions where the sound sources are actually located.

[0140] Therefore, the proposed embodiments aim to improve the perceived quality of spatial audio by combining these two estimates in order to produce a new directional estimate. This approach can exploit the benefits of both estimates (the "direction of arrival estimate of propagating sound" accurately captures the spatial impression, while the "direction estimate of the sound source" accurately captures the direction of the actual sound source). As discussed above, using the new directional estimate to reproduce spatial audio can produce a perception of more point-like and spatially stable sound sources while maintaining the spatial impression of the original directional metadata. In addition, in typical cases, these methods and their implementation devices can improve the intelligibility of speech.

[0141] The spatial audio signal, described in further detail below, may be a microphone array signal, which may be obtained, for example, from a mobile device or a virtual reality camera or any suitable device. The time-frequency domain direction estimate may be analyzed using any suitable method, for example, using spatial audio capture (SPAC) as discussed above.

[0142] Based on any suitable method and using any suitable source data for determination, a wideband direction estimate of the speaker can be obtained. For example, in the following example, the source data is image data obtained from video capture (or camera), and the determination uses a visual source detection method. Examples of visual source detection methods are face detection and tracking. In some other embodiments, a wideband estimate can be obtained based on the use of microphone array signals and auditory source detection methods. In some other embodiments, a wideband estimate can be obtained from metadata that indicates the location of a sound object known to be captured in a spatial audio signal. Other embodiments can be configured to perform object tracking using various other means, such as radio-based indoor positioning or manual tracking by a human operator.

[0143] In some embodiments, the biasing may be implemented such that if the difference between the time-frequency domain directional estimate and the wideband estimate is below a threshold, the time-frequency domain directional estimate is biased towards the wideband estimate. Otherwise, in such embodiments, the original time-frequency domain directional estimate is output or provided without modification. As a result, the directional estimate is more stable and accurate, without changing the general spatial impression of the sound scene.

[0144] In some embodiments, the spatial audio signal may be a transmission signal of a defined number (N) of channels (e.g., two microphone audio signals, stereo audio signals, FOA / HOA, binaural audio signals), and the time-frequency domain direction of arrival estimate is obtained from spatial metadata, which represents the content of the spatial audio scene. In such embodiments, the method may be performed at any stage of the signal processing chain after analysis, such as: within the transmission system, within the transmission mixer, within the early stages of synthesis. In other words, anywhere metadata and broadband estimates are available. In some embodiments, the broadband estimate may also be signaled to the synthesis part of the system via the transmission system.

[0145] about Figure 1 , an example system 101 is shown. The example system 101 includes a capture device 111. In this example, the capture device 111 is a mobile phone (e.g., Nokia 8) that includes two or more microphones configured to generate microphone audio signals 112 (which may also be referred to as microphone array signals) and one or more cameras configured to generate video signals (or image data) 114. In some embodiments, a mobile phone including two microphones can be used to estimate directional parameters (and spatial sound can be reproduced) for an arc of 180 degrees (e.g., in front of the mobile device). In some embodiments where the mobile phone includes three microphones, horizontal directional parameters can be estimated for 360 degrees, while in the case where the mobile phone includes four (or more) microphones, directional parameters can be estimated in 3D (in other words, the directional parameters can include elevation components as well as azimuth components).

[0146] Although the example capture device 111 in these examples is shown as a mobile phone, it will be appreciated that in some embodiments, the capture device 111 is any suitable electronic device or system that includes two or more microphones in embodiments that analyze microphone audio signals to determine time-frequency (TF) domain direction, such as a VR / AR capture device such as the Nokia OZO that includes both a microphone and a camera.

[0147] The microphone array audio signal 112 and the video signal 114 may be passed to an analysis processor 121 .

[0148] In some embodiments, the system 101 includes an analysis processor 121. In some embodiments, the analysis processor 121 can be configured to receive (or otherwise obtain) input signals, such as microphone array audio signals 112 and video signals 114. The analysis processor 121 is configured to generate a transmission audio signal based on the microphone signals and spatial metadata. In some embodiments, the spatial metadata generated by the analysis processor 121 may include directional parameters for frequency bands, and typically include direct to total energy ratio parameters or similar parameters in the frequency bands. In some embodiments, the analysis processor 121 is configured to generate other parameters of the spatial metadata.

[0149] In some embodiments, the analysis processor 121 can be configured to utilize only data from the microphone array signal, in other words, to perform source detection using auditory means rather than visual means. In some embodiments, the analysis processor 121 is configured to utilize information from other sources (such as infrared (IR) or other sensors), or user input (source selection from a touch screen, etc.). In some embodiments, the analysis processor 121 is configured to utilize information from any general tracking service (such as GPS, high-precision indoor positioning (HAIP)). In some embodiments, the analysis processor 121 is configured to utilize location information from metadata associated with audio objects known to be captured in a spatial audio signal. In addition, in some embodiments, the analysis processor 121 is configured to be implemented within an encoder of a codec (such as 3GPP IVAS or MPEG-I), wherein the object direction obtained as input to the encoder is passed to the analysis.

[0150] In some embodiments, the analysis processor 121 is configured to generate a transmission audio signal (e.g., two audio channels) based on the microphone audio signal (in some embodiments, the transmission audio signal can be the microphone audio signal output without any additional processing). In some embodiments, the transmission audio signal and metadata can be passed to the synthesis processor after being transmitted or stored as a data stream 131.

[0151] Transmitted or stored in Figure 1 131. The transmission audio signal and spatial metadata may be encoded, for example, before transmission or storage. For example, the audio signal may be encoded using Advanced Audio Coding (ACC), and the spatial metadata may be encoded in any suitable manner and embedded within the metadata field of the AAC stream. In some embodiments, the encoded audio signal may be sent in a media container with a video stream (such as an encoded video stream based on a video signal generated by a mobile phone). In such an embodiment, when video encoding occurs, the signal and metadata are decoded accordingly before the synthesis processor 141. Some examples of transmission or storage may be:

[0152] Encode the audio and metadata, store it to memory on the mobile phone, and later decode the audio for compositing processing on the same (or different) device;

[0153] Encodes audio and metadata and sends it to a remote device for encoding and compositing processing;

[0154] Feed audio and metadata directly to the compositing processor without encoding;

[0155] The audio and metadata are encoded as described above and sent to a remote device for storage. The remote device makes the encoded audio and metadata available for download by a client. The client downloads packets of the encoded audio and metadata, defragments the downloaded packets, decodes them, and feeds them to the client for synthesis.

[0156] In some embodiments, encoding may be performed in the analysis processor and decoding may be performed in the synthesis processor. If the audio and metadata are provided directly from the analysis processor to the synthesis processor, encoding / decoding may not be necessary.

[0157] In some embodiments, the system 101 includes a synthesis processor 141. The synthesis processor 141 is configured to receive a data stream 122 including a transmission audio signal and metadata. In some embodiments in which the data stream has been encoded and / or multiplexed, the data stream is decoded and / or demultiplexed. In turn, the transmission audio signal and spatial metadata (relating to the direction of the bias processing) are processed (in frequency bands) in a manner that generates a spatialized output (such as a binaural signal). In some embodiments, the synthesis processor 141 is configured to output other spatial audio signal formats, such as Ambisonic signals and stereo or surround speaker signals. In some embodiments such as 360-video playback, the user's head movement can be tracked, and in such embodiments, the synthesis processor 141 is configured to receive head tracking information and generate a suitable output signal based on the head tracking information.

[0158] The synthesis processor 141 may be configured to output the binaural output signals to a suitable output device such as headphones 151. In some embodiments, the headphones 151 or the output device includes a head tracking sensor configured to generate head tracking information that may be communicated back to the synthesis processor 141. The output device (e.g., headphones 151) may be configured to receive output audio signals, such as the binaural output signals 142. The headphones 151, in turn, may be configured to output these audio signals to a listener or user of the headphones 151.

[0159] like Figure 1 The operation of the system shown in Figure 2 An example flow chart is shown in FIG.

[0160] As in Figure 2 As shown in step 200, operation of the system may include obtaining a video signal from a camera.

[0161] As in Figure 2 As shown in step 201, the method may further include obtaining an audio signal from a microphone array.

[0162] As in Figure 2 As shown by step 203, after the video signal and the audio signal have been obtained, the method may include analyzing the video / audio signals to generate a transmission audio signal and spatial audio signal metadata based on the video signal and the audio signal.

[0163] As in Figure 2 As shown by step 205 in FIG. 1 , after the optional storage / transmission operation, the next operation may be to synthesize an output audio signal from the transmission audio signal and the spatial audio signal metadata.

[0164] Furthermore, as in Figure 2 As shown in step 207, these output audio signals are provided to the output device.

[0165] about Figure 3 , showing an example of an analysis processor 121 according to some embodiments.

[0166] In some embodiments, the analysis processor 121 includes an object detector 301. The object detector 301 is configured to receive a video stream (or image data). The object detector can then be configured to find a direction or angle based on the video signal where a source of interest resides. For example, in some embodiments, the object detector 301 can use a method such as described in "You Only Look Once: Unified, Real-Time Object Detection" (Joseph Redmon, Santosh Divvala, Ross Girshick, AliFarhadi, CVPR, 2016). In some embodiments, the object detector 301 is configured to identify a speaker or a person in general using any suitable method.

[0167] In some embodiments, the method may be based on face detection and tracking between adjacent video or image frames. The output of the object detector 301 may be the direction θ of the detected sound source. ss (i,n) 302, where i is the index of the detected sound source and n is the time frame index. There may be any number of detected sound sources, and the number of detected sound sources may change over time, although typically only one or a few sound sources may be detected. In some embodiments, the object detector 301 may not detect any sound source at some time, and the direction information of the detected object may be a null field, or no signal is generated and passed to the spatial audio signal analyzer 303.

[0168] In some embodiments, the analysis processor 121 further includes a transmission audio signal generator 305. The transmission audio signal generator 305 is configured to receive the microphone array audio signal 112 and generate a transmission audio signal 306. In some embodiments, the microphone array audio signal s j (t) (where t is the time sample index and j is the microphone channel index) can be processed by the transmission audio signal generator to form two audio signal channels. For example, in some embodiments, the transmission audio signal generator 305 is configured to generate an audio signal channel with a more prominently presented left sound (relative to the defined front direction) and a second audio signal channel with a more prominently presented right sound. This left-right audio signal configuration can be generated from a microphone array signal, for example, by selecting a suitable microphone pair or by beamforming. The transmission audio signal generator 305 can be further configured to generate a transmission audio signal, which can involve the application of audio signal processing effects, such as equalization, noise suppression, gain control, and limiter processing.

[0169] In some embodiments, the analysis processor 121 further comprises a spatial audio signal analyzer 303. The spatial audio signal analyzer 303 is configured to receive the microphone array audio signal 112 and the direction θ of the detected sound source. ss (i,n) as inputs. The spatial audio signal analyzer 303 is configured to use these inputs to generate metadata 304. The metadata 304 may include one or more directional parameters in the frequency bands, and typically an energy ratio parameter (or several ratio parameters) in the frequency bands, which indicates how much sound energy is directional (relative to each of one or more directions) and how much sound energy is ambient. An example parameterization is a directional parameter θ(k,n) in each frequency band and an associated direct to total energy ratio r(k,n) in each frequency band, where k is the frequency band index and n is the time frame index. In some embodiments, the metadata may be output to a multiplexer 307.

[0170] In some embodiments, the analysis processor 121 includes a multiplexer or MUX 307. The multiplexer 307 is configured to receive the transmission audio signal 306 and the metadata 304. The multiplexer 307 can multiplex the transmission audio signal 306 and the spatial metadata 304 into one data stream and output it as the data stream 122. In some embodiments, the multiplexer 307 is configured to encode the transmission audio signal 306 and the spatial metadata 304, for example in the manner discussed. In some embodiments, the multiplexer 307 is optional or can be configured to be bypassed, and the transmission audio signal 306 and the spatial metadata 304 are output without encoding and / or multiplexing.

[0171] about Figure 4 , showing that Figure 3 An example flow chart of the operation of the analysis processor 121 is shown in FIG.

[0172] As in Figure 4 As shown in step 401, the method may include receiving or obtaining a video / image signal.

[0173] As in Figure 4 As shown in step 403, after the video / image signal has been received or obtained, the next operation is to detect the object from the video / image signal.

[0174] In addition, as in Figure 4 As shown in step 402, in some embodiments, an audio signal is obtained or received.

[0175] As in Figure 4 As shown in step 405, after audio signals have been received or obtained and an object has been detected (and the direction of the object has been determined), the method may include analyzing the audio signals to generate metadata further based on the direction of the detected object.

[0176] In addition, as in Figure 4 As shown in step 404, after the audio signals have been received or obtained from the microphone, a transmission audio signal can be generated from these audio signals.

[0177] As in Figure 4 As shown in step 407, after the transmission audio signal and metadata have been generated, the next operation may be to multiplex these transmission audio signal and metadata into a data stream.

[0178] Furthermore, as in Figure 4 In step 409, the multiplexed signal may be provided as an output for transmission and / or storage.

[0179] about Figure 5 , shows an example spatial audio signal analyzer 303 in more detail. In some embodiments, the spatial audio signal analyzer 303 includes a time to time-frequency domain converter (T to T / F domain converter) 501. The time to time-frequency domain converter 501 is configured to receive the microphone array audio signal s j (t) 112, and transform them into time-frequency S by any suitable time-frequency transform j (k,n) 502. An example of a suitable time-frequency transform is a Short Time Fourier Transform (STFT). In some embodiments, the transformed time-frequency microphone signal 502 is output to a direction / ratio analyzer 503.

[0180] The spatial audio signal analyzer 303 may include a direction / ratio analyzer 503 configured to receive the time-frequency microphone audio signal 502. The direction / ratio analyzer 503 is configured to estimate a preliminary direction θ in the time-frequency domain. pd (k,n) and energy ratio r(k,n). Preliminary direction θ pd (k,n) is an estimate of the main direction of arrival of the propagating sound. The method used to determine the direction and ratio depends on the device type. For example, in some embodiments, the determination or estimation can be a spatial audio capture (SPAC) implementation.

[0181] Thus, for example, in an embodiment implemented based on audio signals captured from a mobile phone, the non-uniform shape of the device means that one of these axes (front-to-back) is typically very thin, which results in little differential information between the microphone signals relative to the front-to-back axis. In other words, this axis has a lower signal-to-noise ratio for parametric analysis than the longer left-to-right axis. Therefore, a suitable option for such a device is to use this information for a binary front-to-back determination, and estimate the direction based on the audio signal from the microphone on the longer axis.

[0182] The process of estimating preliminary directional parameters in the horizontal plane from a device with three microphones can be as follows:

[0183] Estimate the delay parameter d(k,n) in the frequency band between the left-right microphones (of a camera phone in landscape mode);

[0184] Normalize the delay parameter d′(k,n) to between −1 and 1 so that the maximum value is obtained when the sound arrives from the axis determined by the left-right microphone pair;

[0185] An angle parameter indicating a direction between -90 and 90 degrees is obtained by α(k,n)=arcsin(d′(k,n));

[0186] Estimate the delay parameter d in the frequency band between the front and rear microphones (one of which can be the same as the left and right microphones) fb (k,n);

[0187] Find the pre-post delay parameter d fb (k,n) is positive or negative. If the delay parameter indicates that the sound is arriving from behind, then the pd (k,n)=180°-α(k,n) to determine the preliminary direction. Otherwise, use θ pd (k,n)=α(k,n).

[0188] Then, the initial direction θ can be pd (k,n) is passed to a direction bias processor 505. These time-frequency domain direction estimates may be referred to as "parameter-based direction of arrival estimates".

[0189] In some embodiments where the device shape is such that it enables sufficiently robust linear generation of a First Order Ambisonic (FOA) signal, the DirAC method can then be used to generate preliminary directional parameters.

[0190] In some embodiments, the direction / ratio analyzer 503 is configured to generate or estimate an energy ratio parameter based on a normalized (between -1 and 1) cross-correlation parameter c(k,n) between the microphone pair in frequency band k. In some embodiments, a method for determining the energy ratio parameter is to compare the normalized cross-correlation parameter c(k,n) with the diffuse field normalized cross-correlation parameter c(k,n). D (k,n) and compared with

[0191]

[0192] The ratios are determined. These ratios may then be output 504 .

[0193] The spatial audio signal analyzer 303 may include a direction orientation processor 505. The direction orientation processor 505 is configured to receive a preliminary direction θ pd (k,n) and the detected object θ ss Based on these values, the directional bias processor is configured to determine an actual directional parameter θ(k,n) 508 which may be output.

[0194] about Figure 6 , a flowchart illustrating the operation of an example spatial audio signal analyzer 303 is shown.

[0195] As in Figure 6 As shown in step 601, the direction of the detected object is obtained or received.

[0196] In addition, as in Figure 6 As shown in step 602, an audio signal is obtained or received from a microphone array.

[0197] As in Figure 6 As shown in step 604, after the audio signals have been obtained / received from the microphone array, these time domain audio signals are converted into a time-frequency domain audio signal format.

[0198] Furthermore, as in Figure 6 As shown in step 606, these time-frequency domain audio signals are analyzed to determine preliminary direction parameters and ratio parameters.

[0199] As in Figure 6 As shown in step 607, the direction parameter based on the time-frequency domain audio signal and the direction of the detected object are used, and thus, the direction based on the time-frequency domain audio signal can be biased by the direction of the detected object.

[0200] Furthermore, as in Figure 6 As shown in step 609, the deflected direction and ratio parameters are provided.

[0201] about Figure 7 , an example direction bias processor 505 is shown. In some embodiments, the direction bias processor 505 includes a T / F domain direction checker 701. The T / F domain direction checker 701 is configured to receive or obtain the time-frequency domain direction 506 and the direction 302 of the detected object. The T / F domain direction checker 701 is configured to determine or estimate the absolute angular difference between the T / F domain based direction and the object direction. This can be estimated, for example, by the following formula:

[0202] θ ad (k,n,i)=|wrap{θ ss (i,n)-θ pd (k,n)}|

[0203] The |.| notation takes the absolute value, and the wrap{} notation wraps the angle value between -180 and +180 degrees. For simplicity, this example assumes only horizontal directions, but it is straightforward to extend to also have elevation angles, for example if the formulation of the angular difference with elevation angles can be performed using dot products of unit vectors in both directions and arccosine operations.

[0204] In turn, the T / F domain direction checker 701 may be configured to select the smallest of these differences, which may be expressed as:

[0205]

[0206] In addition, the T / F domain direction checker 701 may be configured to select the corresponding signed value θ sd,min (k,n), in other words, information about whether the difference is positive or negative.

[0207] In addition, the T / F domain direction checker 701 is configured to check θ ad,min Is (k,n) less than the threshold θ? T , for example, θ T =30 degrees.

[0208] If the direction checker 701 determines that the smallest of the differences is greater than the threshold, the directions may be passed to a direction bypass 705 .

[0209] In some embodiments, the direction bias processor 505 includes a direction bypass 705 that passes the preliminary time-frequency domain direction unmodified to the output:

[0210] θ(k,n)=θ pd (k,n), when θ ad,min (k,n)≥θ T hour

[0211] It will be understood that direction bypass 705 is optional (in other words, direction checker 701 is configured to output unmodified directions without direction bypass 705). In some embodiments, direction bypass 705 is a buffer configured to store unmodified directions while some other directions are modified by closest object direction deflector 703.

[0212] In some embodiments, the direction deflection processor 505 includes a closest object direction deflector 703. In some embodiments, the closest object direction deflector 703 is configured to be based on the signed value θ sd,min (k,n) and threshold θ T To determine or generate a deflection direction modifier. For example, the closest object direction deflector 703 is configured to determine the deflection direction modifier value based on the following formula:

[0213]

[0214] when , the Bias Direction Modifier value provides an amount to bias the Direction parameter precisely toward the direction of the closest object, and when , interpolation is performed.

[0215] In some embodiments, other equations for determining the bias direction modifier value may be formulated.Thus, if the absolute angle difference is below a threshold, an embodiment biases the preliminary time-frequency direction towards the closest object direction.

[0216] Furthermore, the closest object direction deflector 703 may use the calculated deflection direction modifier value to generate an output time-frequency domain direction, for example, by:

[0217] θ(k,n)=θ pd (k,n)+θ bias (k,n), when θ ad,min (k,n)<θ T hour

[0218] Furthermore, these deflection directions may be output as time-frequency domain directions 508 together with the unmodified time-frequency domain directions.

[0219] In the above examples, the spatial analysis is therefore configured to provide multiple direction estimates or parameterized direction of arrival estimates. In some cases, some of these parameterized estimates may point to reflections of the sound source rather than the actual direction of the sound source. However, the purpose of these embodiments is generally to enable the listener to perceive sounds from the actual direction of the sound source. Therefore, as described, another estimate of the sound source direction (wideband estimate) is obtained, and the direction based on the parameterization (based on the time-frequency audio signal) is modified or biased based on these further estimates. This other estimate should point relatively reliably to the actual main direction of the sound source. Then, it is extended to multiple known sound source directions, and the bias direction is selected from them using the minimum angle difference.

[0220] In some cases, there are two sound sources (A and B) in the room (or in general in the environment), and they have significantly different directions from the listener's perspective. In this case, A and B may be active at the same time, and, for example, the parameterized directional estimates of some reflections of A may come from near the direction of B. This may result in a portion of A's reflections being biased toward B. However, perceptually, this may not be a significant problem, because reflections are not really perceived individually (as sources) in an ordinary room, and a slight mismatch in their directions after biasing will not be very serious. As previously discussed, the directional estimate may deviate from the direction of the actual source for reasons other than due to reflections, for example, due to noise at the microphone (e.g., wind, hand-holding), the acoustic effect of the user's hand near the microphone, the background environment, and reverberation.

[0221] Furthermore, the selection of the minimum angle difference is only an example, and other selection criteria may exist.

[0222] about Figure 8 , showing that Figure 7 The direction shown in the figure is a flowchart of the operation of the processor.

[0223] As in Figure 8 As shown in step 801, the direction of the detected object is obtained or otherwise received.

[0224] In addition, as in Figure 8 As shown in step 802, a direction based on a time-frequency domain audio signal is received or otherwise obtained.

[0225] As in Figure 8 As shown in step 803, after the direction based on the time-frequency domain audio signal and the direction of the detected object have been obtained, the next operation is to determine whether the direction based on the time-frequency domain audio signal is close to any of the object directions.

[0226] As in Figure 8 As shown in step 807, if the difference between the direction based on the time-frequency domain audio signal and any one of the object directions is greater than a threshold, the direction based on the time-frequency domain audio signal is passed on without modification.

[0227] As in Figure 8 As shown in step 805, if the difference between the direction based on the time-frequency domain audio signal and any one of the object directions is less than a threshold, the direction based on the time-frequency domain audio signal is biased toward the closest object direction.

[0228] Furthermore, as in Figure 8 As shown in step 809, biased and / or unmodified directions may be provided.

[0229] about Fig. 9 , shows another example analysis processor 121. In this example, the analysis processor uses only the microphone array audio signal as input.

[0230] In this example analysis processor 121, object detector 901 is configured to receive microphone array signals instead of video streams or image data. Object detector 901 can be configured to determine the direction of a sound source or object based on the microphone array audio signal. For example, in some embodiments, object detector 901 is configured to implement object detection based on a suitable audio detection algorithm, such as described in "Real-Time Multiple Sound Source Localization and Counting Using a Circular Microphone Array" (Despoina Pavlidi, Anthony Griffin, Matthieu Puigt, Athanasios Mouchtaris, IEEE Journal of Audio, Speech and Language Processing, 2013). The output of object detector 901 is the direction θ of the detected sound source. ss (i,n).

[0231] The remaining operations of the analysis processor 121 may be the same as those described above.

[0232] about Fig.10 , showing that Figure 1 An example synthesis processor 141 is shown in FIG.

[0233] In some embodiments, the synthesis processor 141 includes a demultiplexer or DEMUX 1001. The demultiplexer 1001 is configured to receive the data stream 122 and demultiplex and decode the input to generate a transmission audio signal 1006 and metadata (in the form of direction and energy ratio parameters). The transmission audio signal 1006 and metadata 1004 can be provided to the spatial synthesizer 1003.

[0234] In addition, the synthesis processor 141 includes a spatial synthesizer 1003. The spatial synthesizer 1003 is configured to receive the transmission audio signal 1006 and the metadata 1004, and generate a suitable output format audio signal, for example, a binaural output signal 142. The spatial synthesizer 1003 is configured to process the transmission audio signal in a frequency band in a manner that utilizes the metadata to generate a spatialized output such as a binaural signal. In some embodiments, other output spatial audio signal formats that can be generated can be an Ambisonic signal, a stereo or a surround speaker signal. In some embodiments such as 360-video playback, the user's head movement can be tracked, and in this case, the spatial synthesizer is configured to receive and process the transmission audio signal based on the head tracking information input.

[0235] The operation of the spatial synthesizer is known and will not be described in detail.

[0236] about Fig.11 , showing that Fig.10 A flowchart of the operation of the synthesis processor 141 is shown in FIG.

[0237] As in Fig.11 As shown in step 1101, a data stream including a transmission audio signal and metadata is obtained, wherein the metadata includes direction (biased and / or unmodified) and ratio.

[0238] As in Fig.11 As shown in step 1103, after the data stream has been obtained, the data stream is demultiplexed into a transmission audio signal and metadata.

[0239] As in Fig.11 As shown in step 1105, after the transmission audio signal and metadata have been obtained, an output audio signal is generated based on the transmission audio signal and metadata.

[0240] Furthermore, as in Fig.11 As shown by step 1107 , the generated output audio signal may be provided as a synthesized audio signal.

[0241] In some embodiments, the object detector is configured to receive or obtain information other than the video stream or microphone array signal to generate the direction of the detected object. For example, the object detection operation may be affected by user input. In some embodiments, the user can select an object of interest displayed on the touch screen of the camera phone. The phone (or any device with a microphone in an alternative embodiment) may also be equipped with any other sensor to provide information to the object detector for generating the direction of the detected object.

[0242] In some embodiments, the energy ratio parameter can be a direct to total energy ratio, an ambient-to-total energy ratio (diffusivity), a directional stability parameter, or any other suitable ratio. The ratio is configured to indicate how directional or non-directional the sound is for a frequency band.

[0243] In some embodiments, a wideband sound source direction estimate can be obtained from metadata associated with an audio object known to be captured in a microphone array signal. In such embodiments, an audio scene is captured using a microphone array and a close-up signal of an object, thereby providing metadata for the object signal.

[0244] In some further embodiments where object detection is performed on the audio signal (individual audio objects, e.g., close microphones, or spatial audio capture), object detection may be implemented such that an analysis is performed to determine the frequency band in which each object is "located". In some embodiments, this may be achieved, for example, by estimating the energy for each frequency band and then comparing it to a threshold. If the frequency band energy is above the threshold, then it may be determined that the object is "located" within the frequency band (or that the frequency band is active) and used for biasing. If the frequency band energy is below the threshold (or has low energy), then the object is not "located" within the frequency band (or that the frequency band is not active) and should not be used for biasing because it would be biased to an incorrect direction.

[0245] Additionally, in some embodiments, in addition to indicating whether an object is within a frequency band (or whether the frequency band is active), the object detector may also be configured to indicate whether an object is "within" a time frame (or whether the time frame is active), and use this information to decide whether biasing should be performed. In some embodiments, the visual analysis means may provide information on whether a sound source is active, for example, by detecting the movement of a speaker's mouth, where an open or moving mouth may indicate activity, or a closed mouth may indicate that the speaker is not speaking. When object detection is performed based on an audio signal, the time frame detection may be achieved by determining whether the time frame audio signal has energy above a predetermined silence threshold.

[0246] In some embodiments, the object detector is configured to provide additional metadata for the detected sound sources. For example, if a method that provides any confidence information for the direction is used to obtain a wideband direction estimate, the confidence information can be passed to the bias and used to influence the biasing process. Such confidence information can be obtained, for example, by detecting whether a speaker is moving his / her mouth, which corresponds to a high probability that the person is speaking. For example, the confidence information can be used by scaling a threshold so that confident sound source direction estimates are biased more strongly towards these directions and further away than unconfident estimates.

[0247] In some further embodiments, multiple sources of wideband direction estimates (eg, video+audio+radio) may be used together to form a single, more accurate wideband direction estimate that may be used in orientation.

[0248] In further embodiments, biasing may be performed on a Metadata Assisted Spatial Audio (MASA) format (or a similar N channel + metadata format which may be the output of a spatial analysis). In these embodiments, the metadata is modified after analysis by biasing the directional values. For example, this process may be performed on a stored MASA format file within transmission of the format (by decoding and encoding the metadata) or in synthesis before the directional metadata values ​​are used (or during the process of their use). This may be accomplished if a wideband estimate is available for the process. For transmission, this means also sending the wideband estimate to the receiving end, unless the estimate is provided by the receiving end, which allows the receiving user to define the biasing direction.

[0249] Bias processing can be implemented at various stages of processing: in the capture algorithm, in the encoder, in the transmission system, in the transmission mixer, in the decoder, or in the spatial sound renderer. The bias processing can be considered as a separate processing block that can be placed anywhere in the processing chain to enhance directional metadata.

[0250] In some further embodiments, the bias processing can be enabled / disabled by the user. For example, in the example of applying the processing to the MASA format, a typical use case for the format could be mobile phones in future telecommunication networks such as 5G or 6G. In these scenarios, the intelligibility of speech is the most important criterion for the quality of the spatial audio signal. Therefore, if bias is not enabled by default, the user may wish to enable bias manually. In other embodiments, bias may be automatically enabled by the system based on the usage scenario of the parameterized spatial audio: if the spatial audio signal is used for telecommunication purposes (telephone), the system may automatically enable bias. If the spatial audio signal is used for other purposes such as the soundtrack of a video created by the user, the system may provide the user with the option to enable / disable the bias functionality.

[0251] In some embodiments, not all sources / objects need to have wideband direction estimates. Therefore, in some embodiments, a subset of sources / objects have wideband estimates. For example, the present invention enables biasing towards only the most important sources, e.g., the most prominent speakers.

[0252] It should also be noted that although the term "wideband direction estimation" has been used to describe the estimated direction of an object / sound source, in some embodiments and / or situations, it may be used for certain frequencies rather than wideband. For example, it may be known that a certain sound source only has energy at certain frequencies, and in this case, it is useful to apply a bias only at those frequencies.

[0253] Implementation of these embodiments as described above may have advantages such as increasing the accuracy of important sources. In addition, in applications of some embodiments, sources may be made more stable and point-like because they are biased toward known sources. This is particularly evident for devices that often have false direction parameters (e.g., mobile capture analysis). Other advantages may be avoiding changes in other spatial characteristics (e.g., retaining spatiality and envelopment). The intelligibility of speech may be further improved.

[0254] With respect to determining at least one directional parameter of at least one propagating sound associated with at least one audio signal, it is mainly referred to as spatial audio captured with a microphone. However, it should be understood that some embodiments can be implemented in other contexts (e.g., MASA). In addition, with respect to determining at least one direction of at least one sound source associated with at least one audio signal, the "at least one audio signal" is an audio signal that captures the sound source. In other words, the idea is that this direction is associated with a certain sound source in the space where the microphone captures "at least one signal". However, the sound source may not be active all the time. Therefore, for example, if the direction is analyzed using a microphone signal, the sound source direction is an average over a period of time in actual implementation. Therefore, even if he / she does not say anything, or at least during a short break, the sound source direction is basically "always" pointing to the source. In addition, if the sound source detection algorithm is based on, for example, a video, it can detect human faces. In the best case, it should detect when the lips are moving or some other indication that the source is "active" so that when the source is detected, the source is actually speaking. In some embodiments, the device can be configured to detect the face as a "potential" sound source (and accept that it is sometimes wrong). In other words, there may be a source that, at least temporarily, does not emit any sound.

[0255] In such embodiments as discussed above, the estimates are where the device "thinks" the sound sources actually are. Therefore, they may not always generate audio signals.

[0256] The "purpose" of this direction parameter is to point to the sound source that actually emits the sound. If it does not produce any sound at a certain point in time (captured by at least one audio signal), it is more of an "estimation error". In addition, with respect to determining at least one direction of at least one sound source associated with at least one audio signal, the method described herein uses data from different sensors (camera, microphone, IR, or other). Therefore, the actual device or its sensor may have, for example, small calibration problems or non-idealities, which may cause at least one direction of at least one sensed sound source to not accurately point to the actual sound source at the sound scene, but may be offset by 1-5 degrees due to calibration (or other) problems, for example. However, even in this case, the embodiments discussed herein may also attempt to stabilize the direction estimate associated with that source and improve the perceived quality of spatial audio reproduction. In other words, at least one direction of at least one sound source may be more related to the best effort estimate of the sound source direction rather than to the actual sound source direction.

[0257] about Fig.12 , shows an example electronic device that can be used as an analysis or synthesis device. The device can be any suitable electronic device or apparatus. For example, in some embodiments, the device 1700 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc.

[0258] In some embodiments, device 1700 includes at least one processor or central processing unit 1707. Processor 1707 may be configured to execute various program codes, such as the methods described herein.

[0259] In some embodiments, the device 1700 includes a memory 1711. In some embodiments, at least one processor 1707 is coupled to the memory 1711. The memory 1711 can be any suitable storage component. In some embodiments, the memory 1711 includes a program code portion for storing program codes that can be implemented on the processor 1707. In addition, in some embodiments, the memory 1711 can also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). Whenever necessary, the processor 1707 can obtain the implementation program code stored in the program code portion and the data stored in the storage data portion via the memory-processor coupling.

[0260] In some embodiments, the device 1700 includes a user interface 1705. In some embodiments, the user interface 1705 may be coupled to a processor 1707. In some embodiments, the processor 1707 may control the operation of the user interface 1705 and receive input from the user interface 1705. In some embodiments, the user interface 1705 may enable a user to input commands to the device 1700, for example, via a keyboard. In some embodiments, the user interface 1705 may enable a user to obtain information from the device 1700. For example, the user interface 1705 may include a display configured to display information from the device 1700 to the user. In some embodiments, the user interface 1705 may include a touch screen or touch interface that enables information to be input to the device 1700 and also displays information to a user of the device 1700. In some embodiments, the user interface 1705 may be a user interface for communicating with a position determiner as described herein.

[0261] In some embodiments, the device 1700 includes an input / output port 1709. In some embodiments, the input / output port 1709 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 1707 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver device can be configured to communicate with other electronic devices or devices via a wired or wired coupling.

[0262] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as, for example, IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0263] The transceiver input / output port 1709 may be configured to receive signals and, in some embodiments, determine parameters as described herein using the processor 1707 executing appropriate code. In addition, the device may generate appropriate transmission signals and parameter outputs to send to the synthesis device.

[0264] In some embodiments, the device 1700 may be used as at least part of a synthesis device. Thus, the input / output port 1709 may be configured to receive a transmission signal, and in some embodiments receive parameters determined at a capture device or processing device as described herein, and generate a suitable audio signal format output using a processor 1707 executing suitable code. The input / output port 1709 may be coupled to any suitable audio output, such as a multi-channel speaker system and / or headphones (which may be head-tracked or non-tracked headphones), etc.

[0265] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it is well known that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0266] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow as in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip or a memory block implemented within a processor, on a magnetic medium such as a hard disk or a floppy disk, and on an optical medium such as a DVD and its data variant CD.

[0267] The memory may be of any type suitable for the local technical environment and may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit based on a multi-core processor architecture, and a processor.

[0268] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0269] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., can automatically route conductors and position components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0270] The foregoing description has provided a complete and useful description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention as defined by the appended claims.

Claims

1. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least: obtaining at least two audio signals captured by one or more microphones of a device; determining at least one direction of arrival parameter associated with at least one sound source of a sound field based on the at least two audio signals, wherein the at least one direction of arrival parameter at least partially represents a spatial impression of the at least one sound source within the sound field; receiving information provided by a sensor of the device; determining at least one direction of the at least one sound source of the sound field based on the information provided by the sensor; modifying the at least one direction of arrival parameter based on the determined at least one direction; and The modified at least one direction-of-arrival parameter is output.

2. The device according to claim 1, wherein: Causing the device to receive information provided by the sensor includes causing the device to receive information provided by the sensor in response to user input.

3. The device according to claim 2, wherein: Causing the apparatus to receive information provided by the sensor includes causing the apparatus to receive information provided by the sensor in response to a user input selecting an object of interest displayed by the device.

4. The device according to claim 1, wherein: The apparatus is further caused to enable a biasing of the at least one direction of arrival parameter based on the determined at least one direction, and wherein the apparatus is caused to modify the at least one direction of arrival parameter based on the determined at least one direction is dependent on the enabled biasing.

5. The device according to claim 4, wherein: Causing the device to enable the bias includes causing the device to enable the bias in response to a manual input.

6. The device according to claim 4, wherein: Enabling the device to enable the bias includes: enabling the device to automatically enable the bias based on a usage scenario of the spatial audio signal.

7. The device according to claim 6, wherein: Causing the apparatus to automatically enable deflection includes causing the apparatus to automatically enable deflection if the spatial audio signal is used for telecommunication purposes.

8. A method comprising: obtaining at least two audio signals captured by one or more microphones of a device; determining at least one direction of arrival parameter associated with at least one sound source of a sound field based on the at least two audio signals, wherein the at least one direction of arrival parameter at least partially represents a spatial impression of the at least one sound source within the sound field; receiving information provided by a sensor of the device; determining at least one direction of the at least one sound source of the sound field based on the information provided by the sensor; modifying the at least one direction of arrival parameter based on the determined at least one direction; and The modified at least one direction-of-arrival parameter is output.

9. The method according to claim 8, wherein: Receiving information provided by the sensor includes receiving information provided by the sensor in response to user input.

10. The method according to claim 9, wherein: Receiving information provided by the sensor includes receiving information provided by the sensor in response to a user input selecting an object of interest displayed by the device.

11. The method according to claim 8, further comprising: Based on the determined at least one direction, a biasing of the at least one direction of arrival parameter is enabled, wherein based on the determined at least one direction, modifying the at least one direction of arrival parameter depends on the enabled biasing.

12. The method according to claim 11, wherein: Enabling the bias includes: enabling the bias in response to a manual input.

13. The method according to claim 11, wherein: Enabling the bias includes: automatically enabling the bias based on a usage scenario of the spatial audio signal.

14. The method according to claim 13, wherein: Automatically enabling the biasing includes automatically enabling the biasing if the spatial audio signal is used for telecommunication purposes.

15. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: obtaining one or more transmission audio signals based on audio signals captured by one or more microphones of the device; determining at least one direction-of-arrival parameter associated with at least one sound source of a sound field based on the one or more transmitted audio signals, wherein the at least one direction-of-arrival parameter at least partially represents a spatial impression of the at least one sound source within the sound field; receiving information provided by a sensor of the device; determining at least one direction of the at least one sound source of the sound field based on the information provided by the sensor; modifying the at least one direction of arrival parameter based on the determined at least one direction; and The modified at least one direction-of-arrival parameter is output.

16. The device according to claim 15, wherein: The apparatus is further caused to enable a biasing of the at least one direction of arrival parameter based on the determined at least one direction, wherein the apparatus is caused to modify the at least one direction of arrival parameter based on the determined at least one direction depending on the enabled biasing.

17. The device according to claim 16, wherein: Enabling the device to enable the bias includes: enabling the device to automatically enable the bias based on a usage scenario of the spatial audio signal.

18. The device according to claim 17, wherein: Causing the apparatus to automatically enable deflection includes causing the apparatus to automatically enable deflection if the spatial audio signal is used for telecommunication purposes.

19. A method comprising: obtaining one or more transmission audio signals based on audio signals captured by one or more microphones of the device; determining at least one direction-of-arrival parameter associated with at least one sound source of a sound field based on the one or more transmitted audio signals, wherein the at least one direction-of-arrival parameter at least partially represents a spatial impression of the at least one sound source within the sound field; receiving information provided by a sensor of the device; determining at least one direction of the at least one sound source of the sound field based on the information provided by the sensor; modifying the at least one direction of arrival parameter based on the determined at least one direction; and The modified at least one direction-of-arrival parameter is output.

20. The method according to claim 19, further comprising: Based on the determined at least one direction, a biasing of the at least one direction of arrival parameter is enabled, wherein based on the determined at least one direction, modifying the at least one direction of arrival parameter depends on the enabled biasing.

Images