Spatial audio processing

Abstract

The invention relates to spatial audio processing. Examples of the present disclosure relate to spatial audio processing. An apparatus obtains at least two audio signals based on signals from at least two microphones. The apparatus performs, for at least a first frequency range, a first spatial audio processing of the obtained audio signal to generate a first output, and performs, for at least a second frequency range, a second spatial audio processing of the obtained audio signal to generate a second output. The signal processing operation of the first spatial audio processing comprises parameterized audio-based processing, and the signal processing operation of the second spatial audio processing comprises beamformed audio-based processing. The device combines the first output and the second output to generate a combined output.

Description

Technical Field

[0001] Examples of the present disclosure relate to spatial audio processing. Some embodiments relate to spatial audio processing in systems with a limited microphone arrangement. Background Art

[0002] Spatial audio requires audio signals captured by multiple microphones. Some audio capture devices (such as mobile phones) have a limited number of microphones, and the microphones are located in positions that are not optimized for spatial audio capture. These common microphone arrangements may result in quality problems in the captured spatial audio. Summary of the Invention

[0003] According to various but not necessarily all examples of the present disclosure, there is provided an apparatus including components for:

[0004] Obtaining at least two audio signals based on signals from at least two microphones;

[0005] Performing first spatial audio processing on the obtained audio signals for at least a first frequency range to generate a first output;

[0006] Performing second spatial audio processing on the obtained audio signals for at least a second frequency range to generate a second output, wherein the signal processing operations of the first spatial audio processing include processing based on parametric audio, and the signal processing operations of the second spatial audio processing include processing based on beamformed audio; and

[0007] Combining the first output and the second output to generate a combined output.

[0008] The processing based on parametric audio may vary more over time than the processing based on beamformed audio.

[0009] Each output may include at least one of the following:

[0010] Binaural output;

[0011] Multichannel output; and

[0012] Stereo output.

[0013] For at least some frequency ranges in the first frequency range, the second spatial audio processing may not be performed.

[0014] For at least some frequency ranges in the second frequency range, the first spatial audio processing may not be performed.

[0015] The combining may include using the first output for the first frequency range and the second output for the second frequency range.

[0016] The combination may include applying weights to the first output and the second output such that the first output has a higher weight than the second output in a frequency range, and the second output has a higher weight than the first output in another frequency range.

[0017] The component can be used to: determine the orientation of a microphone arrangement and, based on the determined orientation, apply a mode of at least one of the first spatial audio processing or the second spatial audio processing.

[0018] At least one of the first frequency range or the second frequency range may be different for different modes of the first spatial audio processing or the second spatial audio processing.

[0019] The first mode may include using the first spatial audio processing for the first frequency range and using the second spatial audio processing for the second frequency range, and the second mode includes using the first spatial audio processing for both the first frequency range and the second frequency range.

[0020] The first mode may use a first set of coefficients for the second spatial audio processing, and the second mode may use a second set of coefficients for the second spatial audio processing.

[0021] The component can be used to: adjust at least one of the first spatial audio processing or the second spatial audio processing based on the orientation of a listener's head.

[0022] According to various but not necessarily all examples of the present disclosure, there is provided an electronic device including the apparatus as described herein, wherein the electronic device is at least one of the following: a smart phone, a personal computer, an image capture device.

[0023] According to various but not necessarily all examples of the present disclosure, there is provided a method including:

[0024] Obtaining at least two audio signals based on signals from at least two microphones;

[0025] Performing first spatial audio processing on the obtained audio signals for at least a first frequency range to generate a first output;

[0026] Performing second spatial audio processing on the obtained audio signals for at least a second frequency range to generate a second output, wherein the signal processing operations of the first spatial audio processing include processing based on parametric audio, and the signal processing operations of the second spatial audio processing include processing based on beamforming audio; and

[0027] Combine the first output and the second output to generate a combined output.

[0028] According to various but not necessarily all examples of the present disclosure, a computer program including instructions is provided, which when executed by a device cause the device to at least perform:

[0029] Obtain at least two audio signals based on signals from at least two microphones;

[0030] Perform first spatial audio processing on the obtained audio signals for at least a first frequency range to generate a first output;

[0031] Perform second spatial audio processing on the obtained audio signals for at least a second frequency range to generate a second output, wherein the signal processing operations of the first spatial audio processing include processing based on parametric audio, and the signal processing operations of the second spatial audio processing include processing based on beamforming audio; and

[0032] Combine the first output and the second output to generate a combined output.

[0033] Although the above examples and optional features of the present disclosure are described separately, it will be understood that they are provided in all possible combinations and permutations and are included within the present disclosure. It will be understood that various examples of the present disclosure may include any or all of the features described for other examples of the present disclosure, and vice versa. In addition, it will be understood that any one or more or all of the features (in any combination) may be implemented as needed and appropriately by a device, method, and / or computer program instructions, included in a device, method, and / or computer program instructions, and executable by a device, method, and / or computer program instructions. Description of the Drawings

[0034] Some examples will now be described with reference to the drawings, wherein:

[0035] Figure 1 An example electronic device is shown;

[0036] Figure 2 An example method is shown;

[0037] Figure 3 An example processor is shown;

[0038] Figure 4 An example beamforming-based spatial processing block is shown;

[0039] Figure 5 An example parametric-based spatial processing block is shown;

[0040] Figure 6A and Figure 6Billustrates an example beam pattern design; and

[0041] Figure 7 illustrates an example device.

[0042] The drawings are not necessarily to scale. For clarity and conciseness, specific features and views of the drawings may be shown schematically or enlarged in scale. For example, the dimensions of some elements in the drawings may be enlarged relative to other elements to aid in the illustration. Corresponding reference numerals are used in the drawings to label corresponding features. For clarity, not all reference numerals are necessarily shown in all the drawings. Detailed Description

[0043] Figure 1 illustrates an example electronic device 100. The example electronic device 100 may include an audio capture device that may be configured to capture audio of a spatial audio signal. The electronic device 100 may be a smart phone, a personal computer, an image capture device, a teleconferencing device, a capture device integrated into a vehicle, or any other suitable type of device that includes at least two microphones.

[0044] In Figure 1 example, the electronic device 100 includes a processor 102, a memory 104, a plurality of microphones 106, a transceiver 108, and a storage device 110. In Figure 1 only the components of the electronic device 100 mentioned below are shown. The electronic device 100 may include additional components not shown.

[0045] The microphone 106 may include any component that may be configured to detect an audio signal. The microphone 106 provides a microphone audio signal 112 as an output. The microphone audio signal 112 is based on the detected audio signal.

[0046] In Figure 1 example, the electronic device 100 includes three microphones 106. The microphones 100 may be positioned within the electronic device 100 to enable spatial audio capture. In Figure 1 example, a first microphone 106A is disposed at a first end of the electronic device 100, a second microphone 106B is disposed at a second end of the electronic device 100, and a third microphone 106C is disposed at the rear of the electronic device 100. In this example, the microphones 106A to 106C are positioned on a single axis or close to a single axis. Other numbers and arrangements of microphones 106 may be used in other examples of the present disclosure.

[0047] The electronic device 100 is configured such that a microphone audio signal 112 from the microphone 106 is provided as an input to the processor 102. The microphone audio signal 112 can be provided to the processor 102 in any suitable format. In some examples, the microphone audio signal 112 can be provided to the processor 102 in a digital format. The digital format can include pulse code modulation (PCM) or any other suitable type of format. In other examples, the microphone 106 can include an analog microphone 106. In such examples, an analog-to-digital converter can be provided between the microphone 106 and the processor 102.

[0048] The processor 102 can be configured to process the microphone audio signal 112 to perform spatial audio processing of the microphone audio signal 112. Spatial audio processing includes processing the microphone audio signal 112 such that the spatial attributes of the captured audio can be presented. This can enable a listener to perceive the spatial attributes of the audio. For example, the listener can perceive different directions or positions of the audio source.

[0049] The processor 102 can be configured to access program code 116 stored in the memory 104. The processor 102 can use the program code 116 to perform spatial audio processing of the microphone audio signal 112. The processor 102 can use the program code 116 to implement the examples of the present disclosure. The processor 102 can be configured to use the methods as Figures 2 to 5 shown and / or as described herein to perform spatial audio processing.

[0050] The processor 102 processes the microphone audio signal 112 to provide a processed audio signal 114 as an output. Other types of processing can also be applied by the processor 102 to the microphone audio signal 112. For example, the processor 102 can also perform voice enhancement, noise reduction, and / or any other suitable processes.

[0051] The processed audio signal 114 can be provided in any suitable format. The processed audio signal 114 can be provided in an encoded form, such as immersive voice and audio stream (IVAS), binaural audio signal, surround sound speaker signal, ambisonic audio signal, or any other suitable type of spatial audio output. Other formats for the processed audio signal 114 can also be used.

[0052] Figure 1 The illustrated electronic device 100 includes a transceiver 108 and a storage device 110. The processor 102 is coupled to the transceiver 108 and / or the storage device 110 such that the processed audio signal 114 and / or any other output from the processor 102 can be provided to the transceiver 108 and / or the storage device 110.

[0053] The transceiver 108 may include any component that enables data to be transmitted from the electronic device 100. This may enable the processed audio signal 114 and / or any other suitable data to be transmitted from the electronic device 100 to an audio presentation device or any other suitable device.

[0054] The storage device 110 may include any component for storing the processed audio signal 114 from the processor 102 and / or any other output. The storage device 110 may include one or more memories or any other suitable component. The processed audio signal 114 may be retrieved from the storage device 110 at a later time. This may enable the electronic device 100 to present the processed audio signal 114 at a later time, or may enable the processed audio signal 114 to be sent to another device.

[0055] In some examples, the processed audio signal 114 may be associated with other information. For example, the processed audio signal 114 may be associated with an image such as a video image. The image or video image may be captured by a camera of the electronic device 100. The processed audio signal 114 may be associated with other information such that the processed audio signal 114 may be stored with this information and / or may be sent with this other information. This association may enable the image or other information to be provided to a user of the playback device when the processed audio signal 114 is replayed.

[0056] In addition to Figure 1 Devices other than the illustrated electronic device 100 may be used to implement the examples of the present disclosure. For example, in Figure 1 the example, the processor 102 obtains the audio signal 112 from the microphone 106 of the device 100. In other examples, the processor 102 may receive the audio signal 112 from a different source (e.g., the transceiver 108 or the storage device 110).

[0057] Figure 1 The example electronic device 100 includes a common arrangement of microphones 106 that can be used to capture spatial audio. The limited number of microphones 106 and their locations are not ideal for spatial audio capture. For example, there will be a large portion of frequencies for which the arrangement of the microphones 106 is too sparse. In some example devices 100, the arrangement of the microphones 106 may be too small and / or too sparse.

[0058] Parametric spatial audio capture may be used with such an arrangement of microphones 106, however this is not ideal for all cases. For example, if there is a single sound source, parametric spatial audio capture may be used to accurately determine the direction. This results in a stable reproduction of the sound source. However, if there are multiple simultaneously active sound sources, the direction metadata may fluctuate, which may result in a perceivable instability in the reproduction of the sound sources.

[0059] Examples of the present disclosure address these problems and provide a method for spatial audio processing that can provide high-quality, stable reproduction even with a limited microphone arrangement.

[0060] In an example of the present disclosure, the electronic device 100 can operate in multiple orientations. The electronic device 100 is shown in a landscape orientation in Figure 1 . The electronic device 100 can also be used in other orientations, such as a portrait orientation, a main camera orientation, or a selfie camera orientation. Other types of electronic devices 100 can be arranged in different orientations. The methods and processes described herein can be initialized differently for different orientations of the electronic device 100. In some examples, if the electronic device 100 is in a first orientation, the methods described herein can be used, and if the electronic device 100 is in a second orientation, the methods described herein may not be used.

[0061] For example, in the electronic device 100 of Figure 1 , when the electronic device 100 is in a landscape orientation, the microphones 106 are generally in a horizontal plane. When the electronic device 100 is arranged in a portrait orientation, this arrangement of the microphones 106 does not well support horizontal spatial audio capture. Other example electronic devices 100 can include two microphones 106 with appropriate spacing located at one edge of the electronic device 100 (e.g., the right edge of the electronic device 100 as shown in Figure 1 ), such that when the electronic device 100 is in a portrait orientation, the processing according to the present disclosure can also be performed. However, even in this case, the algorithm will be initialized differently because the arrangement and spacing of the microphones 106 will be different according to the orientation.

[0062] Figure 2 An exemplary method is shown. The example method can be implemented by the electronic device 100 as shown in Figure 1 , by the device 700 as shown in Figure 7 or by any other suitable component.

[0063] At block 200, the method includes: obtaining at least two audio signals based on signals from at least two microphones 106. The microphones 106 can be arranged in a normal or restricted arrangement, or can be arranged in any other suitable type of arrangement.

[0064] At block 202, the method includes: performing a first spatial audio processing of the obtained audio signals for at least a first frequency range to generate a first output. At block 204, the method includes: performing a second spatial audio processing of the obtained audio signals for at least a second frequency range to generate a second output.

[0065] The first spatial audio processing and the second spatial audio processing may include different types of processing. The signal processing operations of the first spatial audio processing include processing based on parametric audio. The processing based on parametric audio may include obtaining parameters such as energy and direction parameters for a plurality of frequency bands, and using these parameters and the audio signal to generate a first output.

[0066] The signal processing operations of the second spatial audio processing include processing based on beamformed audio. The processing based on parametric audio varies more over time than the processing based on beamformed audio. The processing based on beamformed audio refers to combining microphone signals, typically on a frequency-by-frequency basis, with time-varying or time-invariant processing weights such that an audio signal with specific spatial capture characteristics is obtained as a result.

[0067] The frequency ranges for which the respective spatial audio processing is used may include a subset of the available frequency range. In such an example, for at least some of the frequency ranges in the first frequency range, the second spatial audio processing is not performed, and / or for at least some of the frequency ranges in the second frequency range, the first spatial audio processing is not performed.

[0068] The frequencies within a frequency range do not need to be continuous. For example, it may be that the second frequency range includes the lowest frequency and the highest frequency, and the first frequency range includes the frequency portion therebetween. In this case, the lowest frequency and the highest frequency will be processed based on beamformed audio, and the frequency portion therebetween will be processed based on parametric audio.

[0069] The frequencies within the respective ranges may depend on the arrangement of the microphone 106, the orientation of the electronic device 100, and / or any other suitable factors. Any suitable procedure may be used to select the frequencies to be used for the respective ranges.

[0070] At block 206, the method includes: combining the first output and the second output to generate a combined output. The combined output may include at least some of the outputs in the first output for the first frequency range and at least some of the outputs in the second output for the second frequency range.

[0071] The combining may include any suitable process. In some examples, the combining may include using the first output for the first frequency range and the second output for the second frequency range. This type of combining may be used if there is no overlap in the respective frequency ranges.

[0072] In some examples, the combination may include applying weights to the first output and the second output. The weights may be applied such that the first output has a higher weight than the second output in a frequency range and the second output has a higher weight than the first output in another frequency range. The weights may be applied such that the first output is the primary or dominant source in one frequency range and the second output is the primary or dominant source in another frequency range. The frequency ranges to which the weights are applied may be the first and second frequency ranges or may be different frequency ranges.

[0073] The combined output may include any suitable type of output. For example, the combined output may include at least one of a binaural output, a multi-channel output, a stereo output, or any other suitable type of output.

[0074] In some examples, the method may include Figure 2 additional blocks not shown in. For example, the spatial audio process may depend on the orientation of the electronic device 100 or the apparatus 700. In such examples, the method may include: determining the orientation of the microphone arrangement and, based on the determined orientation, applying a mode of at least one of a first spatial audio process or a second spatial audio process.

[0075] In some examples, different modes of the corresponding spatial audio process may use different ranges. That is, the first frequency range and / or the second frequency range will be different depending on whether the electronic device 100 or the apparatus 700 is in a landscape orientation or a portrait orientation.

[0076] In some examples, the first mode may be to use one of the corresponding spatial audio processes, while the second mode may be to not use the corresponding spatial audio process. For example, the first mode may include using a first spatial audio process for the first frequency range and a second spatial audio process for the second frequency range, and the second mode may include using the first spatial audio process for both the first frequency range and the second frequency range. In this case, the first spatial audio process will be used for all frequencies, while the second spatial audio process will not be used.

[0077] In some examples, the different modes may include modifications to the corresponding spatial audio process. For example, in the first mode, a first set of coefficients may be used for the second spatial audio process, and in the second mode, a second set of coefficients may be used for the second spatial audio process. The coefficients may be used for a mixing matrix or a processing matrix used in the spatial audio process.

[0078] In some examples, one or more spatial audio processes may be adjusted based on the orientation of the listener's head or any other suitable factor.

[0079] Figure 3Schematically illustrates the operation of an example processor 102 in accordance with an example of the present disclosure. The processor 102 may be included in an electronic device 100 as shown in Figure 1 or may be included in any other suitable type of electronic device 100.

[0080] The processor 102 receives a microphone audio signal 112 as an input. The microphone audio signal 112 may be obtained from two or more microphones 106. The microphones 106 may be arranged in the electronic device 100 as shown in Figure 1 or may be arranged in any other suitable device. In some examples, the microphone audio signal 112 may be retrieved from the memory 104 or may be received from another device.

[0081] The microphone audio signal 112 is provided as an input to a time-frequency transformation block 300. The time-frequency transformation block 300 is configured to transform the input microphone audio signal 112 from a time-domain signal to a time-frequency domain signal. The time-frequency transformation block 300 may be configured to apply any suitable time-frequency transformation to transform the microphone audio signal 112 into a time-frequency audio signal 302. For example, the transformation used by the time-frequency transformation block 300 may include a short-time Fourier transform (STFT), which is configured to take a frame of 1024 samples of the microphone audio signal 112, concatenate the frame with the previous 1024 samples, apply the square root of a 2*1024 length Hann window to the concatenated frame, and apply a fast Fourier transform (FFT) to the result. Other time-frequency transformations and their configurations may be used. The time-frequency transformation block 300 provides the time-frequency audio signal 302 as an output.

[0082] The time-frequency audio signal 302 is provided as an input to both a parametric-based spatial processing block 304 and a beamforming-based spatial processing block 306.

[0083] The parametric-based spatial processing block 304 may be configured to perform processing based on parametric audio. The processing based on parametric audio may include parametric capture and rendering. Parametric capture and rendering may use any suitable process. Parametric capture and rendering may include estimating spatial parameters, such as direction and direct-to-total energy ratio, based on the microphone audio signal 112. These estimated parameters may then be used to synthesize a spatial audio output accordingly. Figure 5 An example of the parametric-based spatial processing block 304 is shown in more detail.

[0084] The processor 102 is configured such that the parametric-based spatial processing block 304 performs processing based on parametric audio for a first frequency range.

[0085] The beamforming-based spatial processing block 306 can be configured to perform processing based on beamformed audio. The processing of the beamformed audio can include rendering based on the measured response or simulated response of the electronic device 100 to generate a beam pattern. In an example where the output processed audio signal 114 is a binaural audio signal, the generated beam pattern can approximate two complex-valued beam patterns (amplitude and phase) of the head-related transfer function related to the left and right ears of a human listener. In other examples, the generated beam pattern can include a beam pattern corresponding to an amplitude panning pattern suitable for stereo reproduction. Figure 4 An example of the beamforming-based spatial processing block 306 is shown in more detail.

[0086] The processor 102 is configured such that the beamforming-based spatial processing block 306 performs processing based on beamformed audio for a second frequency range. The second frequency range can be different from the first frequency range. There may be some overlap between the first frequency range and the second frequency range.

[0087] The frequencies within the frequency range do not need to be continuous. For example, it can be that the second frequency range includes the lowest frequency and the highest frequency, and the first frequency range includes the frequency portion therebetween. In this case, the lowest frequency and the highest frequency will be processed based on beamformed audio, and the frequency portion therebetween will be processed based on parametric audio.

[0088] The frequencies within the corresponding range can depend on the arrangement of the microphones 106, the orientation of the electronic device 100, and / or any other suitable factors.

[0089] The parametric-based spatial processing block 304 provides a first output. The output includes a first frequency portion signal 308. The first frequency portion signal 308 includes a processed time-frequency audio signal based on parametric. The beamforming-based spatial processing module 306 provides a second output. The second output includes a second frequency portion signal 310. The second frequency portion signal 310 includes a processed time-frequency audio signal based on beamforming.

[0090] Both the first frequency portion signal 308 and the second frequency portion signal 310 are provided to a combiner 312. The combiner 312 is configured to combine the first frequency portion signal 308 and the second frequency portion signal 310. The combiner 312 thus combines the first output that has been processed based on parametric audio and the second output that has been processed based on beamformed audio.

[0091] In some examples, the first frequency range and the second frequency range may not overlap. In such examples, the combiner 312 may generate the processed time-frequency audio signal 314 by making frequency-dependent selections of the corresponding inputs. In some examples, there may be some overlap between the first frequency range and the second frequency range. In such examples, the combiner 312 may generate the processed time-frequency audio signal 314 by performing a smooth transition between the corresponding signals. For some frequencies, the combined output may be based on both inputs.

[0092] The combiner 312 provides the processed time-frequency audio signal 314 as an output. The processed time-frequency audio signal 314 is provided to the inverse time-frequency transform block 316.

[0093] The inverse time-frequency transform block 316 is configured to convert the processed time-frequency audio signal 314 into a time-domain signal. The inverse time-frequency transform block 316 may be configured to apply any suitable inverse time-frequency transform to convert the processed time-frequency audio signal 314 into a time-domain signal. The transform used by the inverse time-frequency transform block 316 may be the inverse of the transform used by the time-frequency transform block 300. If the time-frequency transform block 300 uses the STFT, the inverse time-frequency transform block 316 may use the inverse STFT.

[0094] The inverse time-frequency transform block 316 provides the processed PCM audio signal 318 as an output. In Figure 3 the example, the processed PCM audio signal 318 is provided to the encoding block 320. The encoding module 320 is configured to encode the processed PCM audio signal 318. Any suitable program may be used for encoding. For example, Advanced Audio Coding (AAC) or Immersive Voice and Audio Service (IVAS) may be used.

[0095] The encoding block 320 provides the processed audio signal 114 as an output. The processed audio signal 114 may be the output of the processor 102 as shown in Figure 1 . The processed audio signal 114 may be reproduced directly (using headphones or speakers or any other suitable device), and / or they may be stored and / or sent to another device.

[0096] In some examples, the encoding block 320 may be omitted. In such a case, the processed PCM audio signal 318 may be the output of the processor 102.

[0097] Figure 4More specifically, an example beamforming-based spatial processing block 306 is shown. The processing of the beamforming-based spatial processing module 306 includes some offline steps and some online steps. The offline steps need to be performed only once. The offline steps can be performed during the initialization phase of the software that executes the processing according to the present disclosure. The offline steps can even be performed before compiling the processing software so that the program code 116 only includes the resulting beam matrix or other coefficients that will be used during the online processing phase. In other examples, the offline steps can be performed at other times.

[0098] During the offline steps, the target pattern 400 and the device response 404 are provided as inputs to the beam matrix design block 402.

[0099] The target pattern 400 can include a head-related transfer function (HRTF) or any other suitable type of pattern. The HRTF includes a set of complex-valued pairs for different frequencies and different directions. The corresponding HRTF pair includes an amplitude response and a phase response corresponding to the sound arriving at the human ear from a defined direction at a defined frequency.

[0100] The HRTF or other target pattern 400 can be based on measurements (real humans or virtual heads), simulations, or a mixture of measurements and simulations. For example, the target pattern 400 for the lowest frequencies can be obtained from simulations, but the target pattern 400 for higher frequencies can be from measurements.

[0101] The target pattern 400 can include a speaker-based beam pattern. This target pattern 400 can be used when the use case is to capture a multi-channel signal (e.g., stereo) rather than a binaural signal. Details of examples of these speaker patterns are further described below.

[0102] The device response 404 is also provided as an input to the beam matrix design block 402. Similar to the target pattern 400, the device response 404 can include a set of complex weights for each direction and frequency and indicates the phase and amplitude of the sound arriving at the microphone 106 of the electronic device 100 at different directions and frequencies. The device response 404 can also be based on measurements, simulations, or a mixture of measurements and simulations.

[0103] The beam matrix design block 402 is configured to use the input target response 400 and device response 404 to generate a set of coefficients for the beam matrix that is used by the beamforming-based spatial processing block 306.

[0104] The beam matrix design block 402 can use any suitable program. As an example, the target pattern 404 can be represented as t(b,DOA), where t(b,DOA) is a column vector including two (or more) entries, b is the frequency bin index, and DOA is the direction of the arriving sound that can indicate the azimuth and elevation angles. In this example, the target pattern 404 is for binaural or stereo output. Thus, the column vector t(b,DOA) has two rows. The first row is for the left output channel and the second row is for the right output channel.

[0105] For binaural output, the vector t(b,DOA) includes the complex-valued HRTF for bin b and the direction DOA.

[0106] For stereo speaker output, the target pattern 404 can be defined by first converting the direction DOA to a unit vector pointing in the corresponding direction, looking up y(DOA) which is the y-dimension value of this unit vector, and then converting it to the cone-of-confusion azimuth angle value θ cc (DOA) = sin -1 y(DOA). Then, the target pattern is determined such that if θ cc (DOA) ≥ 30°, then and if θ cc (DOA) ≤ -30°, then and when -30° < θ cc (DOA) < 30°; then the t(b,DOA) value is the amplitude translation gain according to a suitable translation scheme (such as the tangent translation law assuming the speakers are at ±30°). In this example of stereo speaker output, the target pattern 404 does not depend on bin b. In some examples, the target pattern 404 for speaker playback can also include a phase that depends on bin b. For example, for the same frequency and direction, the phase of the target value can be the phase of the left and right microphones of the electronic device 100.

[0107] The device response 404 is the same regardless of the processing output. The device response 404 can be represented as a complex-valued column vector r(b,DOA), which includes as many rows as there are microphones 106. In some cases, some of the microphone 106 signals are omitted. For example, the electronic device 100 can include three microphones 106 in the same plane and another microphone 106 in a different plane and above one of the three microphones 106. In such examples, this other microphone 106 can be omitted in the design of the beam matrix.

[0108] The beam matrix design block 402 is configured to design a beam matrix B(b) that aims to optimize the following relationship:

[0109] t(b, DOA) = B(b)r(b, DOA)

[0110] For any response r(b, DOA) from any DOA, the matrix B(b) maps that response to the target response t(b, DOA) in the same direction. Thus, ideally, the matrix B(b) is designed such that sound arriving from any direction is captured as if that capture were according to the target pattern 400.

[0111] The matrix B(b) only approximates this. Such a matrix can only be practically obtained when the wavelength is not too short relative to the spacing of the microphones 106 (due to spatial aliasing) and not too long relative to the spacing of the microphones 106 (due to reduced inter-microphone difference signals and array noise).

[0112] For the electronic device 100 in the lateral mode as Figure 1 shown, the suitable frequency range for obtaining the matrix B(b) is from approximately 0 kHz to approximately 1 kHz or higher. For a smaller microphone array size and correspondingly smaller spacing of the microphones 106, the suitable frequency range can start from a frequency above zero, but can also vary to higher frequencies. When the output is stereo (non-binaural), the suitable frequency range for beamforming audio-based processing is generally smaller because the target pattern 404 of approximate amplitude translation is significantly more spatially selective at the lowest frequencies when compared to the HRTF-based target pattern 404. At the lowest frequencies, the audio wavelength is long relative to the size of the microphone array, and it is no longer feasible to achieve such a spatially selective beam. Thus, for stereo rendering, parametric audio-based processing can be used to render the lowest frequencies.

[0113] The matrix B(b) can be solved by first determining a suitable set of directions DOA where the operations of the matrix are optimized. The operations occurring between these directions are not optimized individually, and thus the number of directions should not be too sparse. The practical choice for optimization is to obtain r(b, DOA) and t(b, DOA) at suitable intervals (e.g., 5-degree intervals represented as DOA1, DOA2, … DOA 72 ) in the horizontal plane, and then stack the data into matrices:

[0114] R(b) = [r(b, DOA1) r(b, DOA2) … r(b, DOA 72 )]

[0115] T(b) = [t(b, DOA1) t(b, DOA2) … t(b, DOA 72 )]

[0116] Then, the solution is:

[0117] B(b) = inverse(R(b)) * T(b)

[0118] Wherein, the operator inverse() can be the Moore-Penrose pseudo-inverse. However, at least some regularization of the inverse is preferred. Omitting the interval index in the symbols, an example is to first use the singular value decomposition R = USV H , and then perform the inverse process on the diagonal matrix s. However, S is not necessarily a square matrix, which depends on the number of microphones 106. Regularized inverse processing can be performed by transposing S and performing operations on all diagonal elements , where s j,j is the element in the j-th row and j-th column of S; s max is the maximum value of s for all j j,j ; and α s is the regularization factor, which can be 0.01. Then, the value replaces all values s of the transposed S o,o , and the result is expressed as Then, the regularized inverse is

[0119] The matrix B(b) is the beam matrix 406 provided as the output of the beam matrix design block 402. The beam matrix 406 is provided as an input to the matrix multiplication block 408. The matrix multiplication block 408 can be performed as part of the online processing.

[0120] The matrix multiplication block 408 also receives the time-frequency audio signal 302 as an input. The time-frequency audio signal 302 can be represented as a column vector s(b,n), which includes as many rows as the number of channels, and where n is the current time frame index (of the time-frequency transform).

[0121] The matrix multiplication block 408 performs the processing by the following operation:

[0122] s b (b,n) = B(b) * s(b,n)

[0123] where the subscript b indicates that the output is the beam-processed output.

[0124] The output of the matrix multiplication block 408 provides the resulting signal s b (b,n) as the output signal. The output signal is the second frequency part 310 as Figure 3 shown.

[0125] The beam matrix 406 can depend on the orientation of the electronic device 100. For example, if the electronic device 100 is in a state such as Figure 1For the horizontal orientation shown, a first set of beam matrices 406 can be determined and used for that orientation. If the microphone array also supports processing in the vertical orientation, a second set of beam matrices 406 can be determined and used for that orientation. Similarly, different beam matrices 406 can be determined based on whether the main camera or the selfie camera of the electronic device 100 is in use.

[0126] The corresponding orientation of the electronic device 100 can be associated with the corresponding beam matrix 406, unless the microphone arrangement has sufficient symmetry such that two or more different orientations match acoustically with each other. In such cases, these orientations of the electronic device 100 can be associated with the same beam matrix 406. In some of these cases, if the left and right directions are different for different orientations, the microphone channels can be switched with each other.

[0127] In the above method, the regularization factor α used by the beam matrix design block 402 s is set to 0.01. The regularization factor can be set to any value between 0 and 1, where a value of 0 means no regularization and 1 means maximum regularization. The value of the regularization factor depends on the use case. A smaller value represents a more accurate approximation of the beam design, but also means higher amplification of potential incoherent noise (e.g., wind noise or microphone noise). For example, the electronic device 100 for outdoor use can have a regularization factor α s = 0.1, while the electronic device 100 for indoor use can have a regularization factor α s = 0.01. The same electronic device 100 can have more than one mode, depending on its usage location. The best mode can be automatically detected (which can be detected based on wind noise detection or camera input or any other suitable input). In an example where a smaller value of the regularization factor is used, for example, if α s = 0.01, the processing can include components for avoiding excessive noise amplification. For example, the input energy and output energy of the beam matrix processing can be monitored, and a limit on the loudness of the output can be imposed. For example, the limiter can be configured such that the average output of the beam matrix design block 402 does not exceed twice the average input energy. This limiter will operate per frequency bin and will remove any overshoots attributed to noise, such as the amplification of microphone noise.

[0128] Figure 5 An example parametric spatial processing block 304 is shown in more detail. In some examples, the parametric processing in the following examples can be replaced by other parametric processing.

[0129] The parametric spatial processing block 304 includes a parametric spatial analysis block 500 and a parametric spatial synthesis block 504.

[0130] The parametric spatial analysis block 500 receives the time-frequency audio signal 302 and uses the time-frequency audio signal 302 to determine the spatial metadata 502. The spatial metadata 502 may include direction parameters and direct total energy ratio parameters for a frequency band k. A frequency band k may include multiple frequency bins b, and have a lowest frequency bin b low (k) and a highest bin b high (k). The frequency resolution of these frequency bands may be a frequency resolution following the Bark frequency scale or any other suitable frequency resolution. A typical practical design principle is that for higher frequency bands k, there are a higher number of bins b included within these bands.

[0131] Any suitable process may be used to determine the spatial metadata 502. In some examples, the spatial metadata 502 may be determined as follows. In this example, only a subset of the available microphone audio signals 112 is used to determine the spatial metadata 502. For example, only the microphone audio signal 112 from the microphones 106 at the edges of the electronic device may be used.

[0132] s(b,n,1) and s(b,n,2) may represent the first and second row entries of s(b,n). The cross-correlation may be represented by the following formula:

[0133] c(b,n) = s(b,n,1)s * (b,n,2)

[0134] Then, for a delay d, the real-valued normalized correlation metric in the range from bin b0 to b1 may be defined by the following formula:

[0135]

[0136] The delay search range may be selected based on the spacing of the microphones 106. The distance between the first microphone 106A and the second microphone 106B may be represented as d mic . This distance may be 0.16 meters. Then, the search may be performed by: determining 41 evenly spaced azimuth search directions θ d , which range from -90° to 90°; and then by converting them to delay values, where v s is the speed of sound.

[0137] For a frequency band k, for each d = d(θ d ), the following value is evaluated:

[0138] c norm,k (k,d,n) = c norm (b low(k), b high (k), d, n)

[0139] Then, for each frequency band k, for d = d(θ d ), the value (for which the maximum value c norm,k (k, d, n)) is obtained and evaluated. This value of d is the delay estimate obtained for that frequency band. Thus, for that frequency band, the determined direction parameter θ(k, n) is then the corresponding θ d , and for the same d, the total ratio parameter r(k, n) is directly the value of c norm,k (k, d, n). This process is repeated for each frequency band k.

[0140] In this example, the direction parameter is only the azimuth angle, and no elevation angle analysis is performed. The elevation angle can be set to zero. In other examples, the elevation angle can also be estimated. Any suitable method can be used to estimate the elevation angle.

[0141] The determined direction parameter θ(k, n) and ratio r(k, n) provide spatial metadata 502. The spatial metadata 502 is provided as the output from the parametric spatial analysis block 500.

[0142] In the described example, the processing of the parametric spatial analysis block 500 is based on the delay estimate between two microphones 106 at the edge of the electronic device 100 to provide the azimuth direction parameter θ(k, n). The angle is assumed to be in the front horizontal plane. This means that any sound arriving from the rear direction is mirrored to the front. In some examples, the parametric capture can resolve whether the sound arrives from the front or the rear. If it is determined that the sound arrives from the rear, the azimuth direction parameter θ(k, n) is mirrored to the rear (for example, a sound at 30 degrees is mirrored to 150 degrees).

[0143] Any suitable process can be used to determine whether the azimuth direction parameter θ(k, n) should be in the front or the rear. In Figure 1 the example electronic device 100, this process can include using the delay analysis between the second microphone 106B and the third microphone 106C to determine whether the sound at the time-frequency interval (k, n) is in the front or the rear. However, for binaural rendering, performing such front-rear determination is not crucial because the binaural cues are similar for the front and rear directions within the same cone of confusion. Additionally, for stereo playback, the rear direction is mirrored to the front.

[0144] In Figure 5In the example of, spatial metadata is determined based on the parameterized spatial processing block 304. In some examples, the spatial metadata 502 may have been previously formulated such that the parameterized spatial processing block 304 receives the spatial metadata 502 as an input. The previously formulated spatial metadata 502 can be formulated using any suitable procedure. In such an example, the parameterized spatial processing block 304 need not include the parameterized spatial analysis block 500.

[0145] The parameterized spatial synthesis block 504 receives the time-frequency audio signal 302, the target pattern 400, and the spatial metadata 502 as inputs. The parameterized spatial synthesis block 504 is configured to render a spatial audio output based on the respective inputs. Any suitable process can be used to render the spatial audio output.

[0146] As an illustrative example of using Figure 1 the electronic device 100 for binaural rendering, the parameterized spatial synthesis block 504 may only perform level manipulation. The time-frequency audio signal 302 used may only include signals from the left microphone 106A and the right microphone 106B. The energy or level of these signals is modified based on the spatial metadata 502. In this example configuration, the spacing between the microphones 106A, 106B is similar to the spacing of human ears (and thus, the interaural time difference is correct or nearly correct), but the head shadow effect is incorrect. Therefore, the parameterized spatial synthesis block 504 will modify the spectral cues.

[0147] For each frequency band k, the input energy is formulated for channels i = 1, 2 by the following formula:

[0148]

[0149] Then, the target energy is formulated using the spatial metadata 502 and the target pattern 400. The target pattern t(k,DOA) is available in the frequency band k. The target pattern 400 used by the parameterized spatial synthesis block 504 may be the same as the target pattern 400 used by the beamforming-based spatial processing block 306, or may be derived from the same dataset used for the target pattern used by the beamforming-based spatial processing block 306, or may be derived from a different dataset.

[0150] A pattern with azimuth angle θ(k,n) and zero elevation angle is selected from the target pattern t(k,DOA). If there are no exactly available data points in the target pattern 400, the nearest DOA may be selected or interpolation may be performed. The selected target pattern vector is denoted as t(k,θ(k,n)). The target energy for the frequency band k is then defined by the following formula:

[0151] E out (k,n,i)=(t(k,θ(k,n),i) 2r(k,n)+0.5(1 - r(k,n)))(E in (k,n,1)+E in (k,n,2))

[0152] where t(k, θ(k,n), i) is the i-th row value of t(k, θ(k,n)). In this example, assuming the case of HRTF reproduction, the target pattern t(k, θ(k,n)) is diffuse field equalized such that its average value in the surrounding space directions is

[0153] Then, the parameterized spatial audio output signal is obtained by the following formula:

[0154] s p (k,n,i) = g(k,n,i)s(k,n,i)

[0155] where

[0156]

[0157] The division for obtaining the energy of g(k,n,i) can be regularized by a small value (epsilon) at the divisor and the maximum value of g(k,n,i) can be limited to a value such as 4.

[0158] The signal s p (k,n,i) is represented in vector form as s p (b,n). This signal provides the first frequency portion 308 as shown, and can be provided as the output of the parameterized spatial processing block 304. In the example of the present disclosure, only a subset of the interval of s Figure 3 (b,n) needs to be formulated because a portion of the frequencies can be formulated by the beamforming-based spatial processing block 306. p (b,n).

[0159] In the above example, binaural rendering can be used. Other types of spatial audio outputs can be used. For stereo rendering, the above example can be appropriately modified, or different methods can be used.

[0160] The signal s p (b,n) provides the first frequency portion 308 that has been processed based on parameterized audio, while the signal s b (b,n) provides the second frequency portion 310 that has been processed based on beamforming audio. These corresponding signals can be combined by the combiner 312. The combiner 312 can use any suitable process. In some examples, the combiner 312 can take intervals from different inputs such that the combined output is s b(b,n), and the combination output for intervals where beamformed audio-based processing is not feasible is s p (b,n). In some examples, the combination output can be interpolated at some frequencies such that for at least some frequencies, the combination output can include a mix s b (b,n)m(b)+s p (b,n)(1 - m(b)), where m(b) is a value that transitions from 0 to 1 or from 1 to 0 during the interpolation frequency interval.

[0161] In some examples of the present disclosure, the parametric-based spatial processing block 304 can perform parametric audio-based processing only for a first frequency range, while the beamforming-based spatial processing block 306 can perform beamformed audio-based processing only for a second frequency range. In some examples, either or both of parametric audio-based processing or beamformed audio-based processing can also be performed outside the allocated frequency range.

[0162] For example, in some examples, parametric audio-based processing can be performed for the entire frequency range. In such a case, the processed audio signal 114 can be encoded with spatial metadata 502 for the entire frequency range. This can enable the spatial metadata 502 to be used at a later stage and / or at a remote decoder to present the processed audio signal 114 in other formats.

[0163] In some cases, head tracking can be utilized to implement examples of the present disclosure. In such a case, the beamforming-based spatial processing block 306 will determine a beam matrix 406 for different head orientations. Different head orientations can include orientations with a 5-degree spacing for yaw rotation. The beamforming-based spatial processing block 306 can use the methods described above or any other suitable method, but the target pattern 400 will rotate based on the orientation of the head.

[0164] In such examples, the beam matrix 406 will be used by the matrix multiplication block 408 as part of the online processing. To accommodate head tracking, the correct beam matrix 406 will be determined for the current head orientation. The correct beam matrix 406 can be selected by choosing the nearest beam matrix 406 from the determined matrices or by interpolation.

[0165] The parametric-based spatial processing block 304 can account for the head orientation by rotating the determined direction parameter θ(k,n) before applying the determined direction parameter θ(k,n) in the rendering. In some examples, the parametric-based spatial processing block 304 can also perform some additional processing of the audio signal based on the head orientation before the rendering.

[0166] Thus, examples of the present disclosure can be used with head tracking. Head-tracked binaural rendering can be performed by processing the stored or acquired microphone audio signal 112 in real time based on the head orientation of the tracked listener.

[0167] In examples using head tracking, the frequency range for a corresponding type of spatial processing can change over time based on the head orientation of the listener and the position of the microphones 106 in the electronic device 100. For example, the crossover frequency between a first frequency for which processing based on parametric audio is used and a second frequency for which processing based on beamformed audio is used can be changed. The change in the frequency range can take into account that the suitable frequency range for processing based on beamformed audio can be different for different head orientations.

[0168] Although the examples described above have been described for binaural and stereo outputs, other types of outputs can be used in other examples. Examples of the present disclosure can be used to perform processing for any multi-channel output, such as a 5.1 multi-channel speaker setup. The processing steps can be as described above, where processing based on beamformed audio is used for the frequency range in which a suitable beam pattern can be achieved with the microphones 106 in use, and processing based on parametric audio is used at other frequencies.

[0169] In examples where the output is a multi-channel output (e.g., a 5.1 multi-channel output), there are more output channels than would be used for a binaural or stereo output. The beam pattern for a multi-channel output is narrower than the beam pattern for a binaural or stereo output. This means that more microphones 106 may be required to obtain the device response 404 for such a speaker setup. Thus, to implement examples of the present disclosure for multi-channel output, the electronic device 100 may require more than three microphones 106. For example, the electronic device 100 may require five to ten microphones 106.

[0170] Examples of the present disclosure can also be implemented with other types of signals. For example, examples of the present disclosure can be applied with crosstalk cancellation stereo reproduction. Crosstalk cancellation stereo can be used in an electronic device (e.g., a mobile phone) having stereo speakers. Crosstalk cancellation stereo reproduction is applied to create the perception of a stereo image that is wider than the physical position of the speakers. With such a reproduction system, the processing will be as described above, but in the processing based on beamformed audio, the target pattern 400 for normal stereo reproduction will be replaced by the corresponding target pattern 400 for crosstalk cancellation stereo reproduction.

[0171] Examples of the present disclosure may also be implemented using a soundbar reproduction system. In such an example, the device response 404 from the soundbar will be used for beamforming audio-based processing. Any suitable process may be used to perform parametric audio-based processing.

[0172] Figure 6A and Figure 6B show an example beam pattern design for approximating HRTF responses. The electronic device 100 for obtaining the beam pattern may be a mobile phone or a device of a similar shape. The electronic device 100 can be arranged in a landscape mode.

[0173] Figure 6A show an example beam pattern design for an electronic device 100 having two microphones 106. The two microphones 106 may be located at the left and right edges of the electronic device 100. Figure 6B show an example beam pattern design for an electronic device 100 having six microphones 106. The arrangement of the six microphones may include three microphones 106 located at the left edge of the electronic device 100, three microphones 106 located at the right edge of the electronic device 100, and additional microphones 106 may be located on the front and rear surfaces near the edges.

[0174] In Figure 6A and 6B , the solid lines represent the beam-based HRFT capture patterns obtained using the respective microphone arrangements, while the dashed lines represent the actual HRTF capture patterns in the horizontal plane. The beam matrix may be designed as described above.

[0175] Figure 6A and Figure 6B show the corresponding patterns for the left and right channels at multiple different frequencies. The frequencies shown are 211 Hz, 563 Hz, 1852 Hz, and 7008 Hz.

[0176] Figure 6A and 6B show a single frequency where the beamforming audio-based processing no longer well approximates the target pattern. Instead, the pattern gradually becomes an aliased pattern. This may affect the frequencies selected for the respective types of processing. In some examples, a suitable selection for a first frequency range (for parametric audio-based processing) and a second frequency range (for beamforming audio-based processing) would be to make the first range as frequencies above 1 kHz and the second range as frequencies below 1 kHz.

[0177] The frequencies that will be used for the respective ranges may be determined by comparing with Figure 6A and 6BSelected by examining the obtained beam pattern against the target pattern shown. From this examination, the frequencies at which the beamformed-audio-based processing sufficiently approximates the target pattern can be determined. These frequencies can then be used as the second frequency range (for beamformed-audio-based processing), while the other frequencies can be used as the first frequency range (for parameterized-audio-based processing). This method of selecting the frequency ranges can be used for any arrangement of microphones, orientation of the electronic device 100, and target pattern.

[0178] In Figure 6A and Figure 6B example, the obtained beam capture pattern can approximate the HRTF pattern at low frequencies. At 563 Hz, the microphone arrangement with six microphones 106 performs slightly better than the microphone arrangement with two microphones 106. At higher frequencies, both arrays significantly suffer from spatial aliasing. Thus, at high frequencies, beamformed-audio-based processing is not feasible for these form factors and microphone arrangements.

[0179] However, Figure 6A and Figure 6B show that beamformed-audio-based processing is feasible at low frequencies. This means that parameterized-audio-based processing can be replaced by beamformed-audio-based processing at low frequencies. This provides the following benefits: for the selected frequency ranges, regardless of how complex the sound scene is, beamformed-audio-based processing represents the spatial sound scene well and does not have the spatial capture artifacts (e.g., source instability) that may occur in parameterized-audio-based processing.

[0180] Figure 7 Schematically shows a device 700 that can be used to implement an example of the present disclosure. The device 700 includes at least one processor 102 and at least one memory 104. The processor 102 and the memory 104 can be as Figure 1 shown. The device 700 can include Figure 7 additional components not shown in

[0181] In Figure 7 example, the device 700 includes processing means. The device 700 can be configured to process audio data. The device 700 can be configured to process audio data for use in a communication system or for any other suitable purpose.

[0182] In Figure 7In the example, the implementation of apparatus 700 can be a processing circuit. In some examples, apparatus 700 can be implemented only in hardware, only have specific aspects in software (including firmware), or can be a combination of hardware and software (including firmware).

[0183] As Figure 7 shown in, apparatus 700 can be implemented using instructions that implement hardware functions, such as by using executable instructions of computer program 116 in general-purpose or special-purpose processor 102, and these executable instructions can be stored on a computer-readable storage medium (such as a disk, memory, etc.) for execution by such processor 102.

[0184] Processor 102 is configured to read from and write to memory 104. Processor 102 may further include: an output interface through which processor 102 outputs data and / or commands; and an input interface through which data and / or commands are input into processor 102.

[0185] Memory 104 is configured to store computer program 116 including computer program instructions (computer program code 704), and when these computer program instructions are loaded into processor 102, they control the operation of apparatus 700. The computer program instructions of computer program 116 provide the logic and routines that enable apparatus 700 to execute the methods described herein. Processor 102 can load and execute computer program 116 by reading memory 104.

[0186] Apparatus 700 thus includes: at least one processor 102; and at least one memory 104 storing instructions that, when executed by at least one processor 102, cause apparatus 700 to at least perform:[[]]

[0187] Based on signals from at least two microphones, obtain at least two audio signals;

[0188] For at least a first frequency range, perform first spatial audio processing on the obtained audio signals to generate a first output;

[0189] For at least a second frequency range, perform second spatial audio processing on the obtained audio signals to generate a second output, wherein the signal processing operations of the first spatial audio processing include processing based on parametric audio, and the signal processing operations of the second spatial audio processing include processing based on beamforming audio; and

[0190] Combine the first output and the second output to generate a combined output.

[0191] As Figure 7As shown, the computer program 116 can reach the device 700 via any suitable transmission mechanism 910. The transmission mechanism 702 can be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a storage device, a recording medium (such as a compact disc read-only memory (CD-ROM) or a digital versatile disc (DVD) or a solid-state memory), an article of manufacture including or tangibly embodying the computer program 116. The transmission mechanism can be a signal configured to reliably transmit the computer program 116. The device 700 can propagate or transmit the computer program 116 as a computer data signal. In some examples, a wireless protocol (such as Bluetooth, Bluetooth Low Energy, Bluetooth Smart, 6LoWPan (Low-Power Personal Area Network IPv6), ZigBee, ANT+, near field communication (NFC), radio frequency identification, wireless local area network (Wireless LAN) or any other suitable protocol) can be used to send the computer program 116 to the device 900.

[0192] The computer program 116 can include computer program instructions for causing the device 700 to at least perform the following operations or for at least performing the following operations:

[0193] Based on signals from at least two microphones, obtain 200 at least two audio signals;

[0194] For at least a first frequency range, perform 202 first spatial audio processing on the obtained audio signals to generate a first output;

[0195] For at least a second frequency range, perform 204 second spatial audio processing on the obtained audio signals to generate a second output, wherein the signal processing operations of the first spatial audio processing include processing based on parametric audio, and the signal processing operations of the second spatial audio processing include processing based on beamforming audio; and

[0196] Combine 206 the first output and the second output to generate a combined output.

[0197] The computer program instructions can be included in the computer program 116, a non-transitory computer-readable medium, a computer program product, a machine-readable medium. In some but not necessarily all examples, the computer program instructions can be distributed over multiple computer programs 116.

[0198] Although the memory 104 is shown as a single component / circuit, it can be implemented as one or more separate components / circuits, some or all of which can be integrated / removable, and / or can provide permanent / semi-permanent / dynamic / cache storage.

[0199] Although the processor 102 is shown as a single component / circuit, it may be implemented as one or more separate components / circuits, some or all of which may be integrated / movable. The processor 102 may be a single-core or multi-core processor.

[0200] References to "computer-readable storage media", "computer program products", "tangibly embodied computer programs", etc. or "controllers", "computers", "processors", etc. should be understood to include not only computers having different architectures (e.g., single / multi-processor architectures and sequential (von Neumann) / parallel architectures), but also dedicated circuits such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc. should be understood to include software or firmware for programmable processors, such as programmable content for hardware devices, whether instructions for a processor or configuration settings for fixed function devices, gate arrays, or programmable logic devices, etc.

[0201] As used in this application, the term "circuit" may refer to one or more or all of the following:

[0202] (a) A pure hardware circuit implementation (e.g., an implementation in pure analog and / or digital circuits), and

[0203] (b) A combination of hardware circuits and software, e.g., (where applicable):

[0204] (i) A combination of analog and / or digital hardware circuits with software / firmware, and

[0205] (ii) Any part of a hardware processor (including a digital signal processor), software, and memory that work together to cause a device (e.g., a mobile phone or a server) to perform various functions, and

[0206] (c) A hardware circuit and / or processor, such as a microprocessor or a part of a microprocessor, that requires software (e.g., firmware) to operate, but the software may not be present when not needed for operation.

[0207] This definition of "circuit" applies to all uses of the term in this application (including in any claims). As a further example, as used in this application, the term "circuit" also encompasses an implementation of a pure hardware circuit or a processor with its attendant software and / or firmware. The term "circuit" also encompasses (e.g., and if applicable to a particular claim element) a baseband integrated circuit for a mobile device or a similar integrated circuit in a server, a cellular network device, or other computing or network device.

[0208] The boxes shown in the accompanying drawings may represent steps in a method and / or segments of code in a computer program 116. The illustration of a particular order of the boxes does not necessarily imply that the boxes have a required or preferred order, and the order and arrangement of the boxes may be changed. Additionally, some boxes may be omitted.

[0209] In Figure 7 the example of, apparatus 700 is shown as a single entity. In other examples, apparatus 700 may be provided as multiple distinct entities that may be distributed within a cloud or other suitable network.

[0210] As used in this document, the term "comprising" has an inclusive meaning rather than an exclusive meaning. That is, any reference to X that comprises Y indicates that X may comprise only one Y or may comprise multiple Ys. If the intention is to use "comprising" with an exclusive meaning, it will be made explicit in the context by reference to "comprising only one" or by using "consisting of".

[0211] In this specification, the phrases "connected", "coupled", and "communicating" and their derivatives refer to being connected / coupled / communicating operationally. It should be understood that any number or combination of intermediate components (including no intermediate components) may exist, i.e., so as to provide a direct or indirect connection / coupling / communication. Any such intermediate components may include hardware and / or software components.

[0212] As used herein, the term "determine" (and its grammatical variants) may include, but is not limited to: calculating, estimating, processing, deriving, measuring, investigating, identifying, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), obtaining, etc. Additionally, "determine" may include parsing, selecting, picking, establishing, etc.

[0213] In this specification, various examples have been referred to. The description of a feature or function with respect to an example indicates that these features or functions exist in that example. The use of the term "example" or "for example" or "able to" or "may" in this document (whether explicitly stated or not) indicates that such a feature or function exists at least in the described example (whether described as an example or not), and they may but do not necessarily exist in some or all other examples. Thus, "example", "for example", "able to", or "may" refer to a particular instance in a class of examples. The characteristics of an instance may be the characteristics of only that instance or the characteristics of the class or a subclass of the class (which includes some but not all of the instances in the class). Thus, it is implicitly disclosed that features described with reference to one example and not with reference to another example may (if possible) be used as part of a working combination in that other example, but do not necessarily have to be used in that other example.

[0214] Although examples have been described in the preceding paragraphs with reference to various examples, it should be understood that the given examples can be modified without departing from the scope of the claims.

[0215] The features described in the foregoing description can be used in combinations other than those explicitly described above.

[0216] Although functions have been described with reference to specific features, these functions can be performed by other features, whether or not described.

[0217] Although features have been described with reference to specific examples, these features can also be present in other examples, whether or not described.

[0218] The term "a" or "the" as used in this document has an inclusive rather than an exclusive meaning. That is, any reference to X that includes a / the Y indicates that X can include only one Y or can include multiple Ys, unless the context clearly indicates otherwise. If the intention is to use "a" or "the" with an exclusive meaning, it will be made explicit in the context. In some cases, "at least one" or "one or more" may be used to emphasize the inclusive meaning, but it should not be assumed that any exclusive meaning can be inferred without these terms.

[0219] The feature (or combination of features) present in a claim refers to the feature (or combination of features) itself and also to features (equivalent features) that achieve substantially the same technical effect. Equivalent features include, for example, features that are variants and that achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0220] In this specification, adjectives or adjective phrases have been used to describe the characteristics of examples with reference to various examples. Such a description of a characteristic with respect to an example indicates that the characteristic is present in some examples exactly as described and in other examples substantially as described.

[0221] The foregoing description has described some examples of the present disclosure, but those of ordinary skill in the art will know of possible alternative structural and method features that provide functions equivalent to those of the specific examples of such structures and features described above and that have been omitted from the foregoing description for the sake of brevity and clarity. However, the foregoing description should be understood to implicitly include a reference to such alternative structural and method features that provide equivalent functions, unless such alternative structures or method features are explicitly excluded in the foregoing description of the examples of the present disclosure.

[0222] When, in the foregoing specification, attention has been directed to those features regarded as important, it should be understood that the applicant may seek protection, by way of claim, for any patentable feature or combination of features mentioned above and / or shown in the drawings, whether or not such features are emphasized.

Claims

1. A device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: obtaining at least two audio signals based on signals from at least two microphones; performing, for at least a first frequency range, a first spatial audio processing of the obtained audio signal to generate a first output; performing, for at least a second frequency range, a second spatial audio processing of the obtained audio signal to generate a second output, wherein the signal processing operations of the first spatial audio processing include parametric audio based processing and the signal processing operations of the second spatial audio processing include beamformed audio based processing; as well as The first output and the second output are combined to generate a combined output.

2. The device according to claim 1, wherein: The apparatus is configured such that the parametric audio-based processing varies more over time than the beamformed audio-based processing.

3. The device according to claim 1, wherein: Each output includes at least one of the following: Binaural output; Multi-channel output; and Stereo output.

4. The device according to claim 1, wherein: For at least some frequency ranges of the first frequency ranges, the second spatial audio processing is not performed.

5. The device according to claim 1, wherein: For at least some frequency ranges of the second frequency range, the first spatial audio processing is not performed.

6. The device according to claim 1, wherein: Such that the device combines at least one of the following: using the first output for the first frequency range and the second output for the second frequency range; or Weighting is applied to the first output and the second output such that the first output has a higher weighting than the second output in a frequency range and the second output has a higher weighting than the first output in another frequency range.

7. The device according to claim 1, wherein: The apparatus is caused to: determine an orientation of the microphone arrangement, and based on the determined orientation, apply a mode of at least one of the first spatial audio processing or the second spatial audio processing.

8. The device according to claim 7, wherein: At least one of the first frequency range or the second frequency range is different for different modes of the first spatial audio processing or the second spatial audio processing.

9. The device according to claim 7, wherein: A first mode causes the device to use the first spatial audio process for the first frequency range and a second spatial audio process for the second frequency range, and a second mode causes the device to use the first spatial audio process for both the first frequency range and the second frequency range.

10. The device according to claim 7, wherein: The first mode causes the apparatus to use a first set of coefficients for the second spatial audio processing and the second mode causes the apparatus to use a second set of coefficients for the second spatial audio processing.

11. The device according to claim 1, wherein: The apparatus is caused to adjust at least one of the first spatial audio processing or the second spatial audio processing based on a head orientation of a listener.

12. A method comprising: obtaining at least two audio signals based on signals from at least two microphones; performing, for at least a first frequency range, a first spatial audio processing of the obtained audio signal to generate a first output; performing, for at least a second frequency range, a second spatial audio processing of the obtained audio signal to generate a second output, wherein the signal processing operations of the first spatial audio processing include parametric audio based processing and the signal processing operations of the second spatial audio processing include beamformed audio based processing; as well as The first output and the second output are combined to generate a combined output.

13. The method according to claim 12, wherein: The parametric audio based processing varies more over time than the beamformed audio based processing.

14. The method of claim 12, further comprising at least one of the following: for at least some of the first frequency ranges, not performing the second spatial audio processing; or For at least some frequency ranges of the second frequency range, the first spatial audio processing is not performed.

15. The method according to claim 12, wherein: The combination includes at least one of the following: using the first output for the first frequency range and the second output for the second frequency range; or Weighting is applied to the first output and the second output such that the first output has a higher weighting than the second output in a frequency range and the second output has a higher weighting than the first output in another frequency range.

16. The method according to claim 12, further comprising: An orientation of the microphone arrangement is determined, and based on the determined orientation, a mode of at least one of the first spatial audio processing or the second spatial audio processing is applied.

17. The method according to claim 16, wherein: At least one of the first frequency range or the second frequency range is different for different modes of the first spatial audio processing or the second spatial audio processing.

18. The method according to claim 16, wherein: The first mode comprises using the first spatial audio process for the first frequency range and using the second spatial audio process for the second frequency range, and the second mode comprises using the first spatial audio process for both the first frequency range and the second frequency range.

19. The method according to claim 16, wherein: The first mode uses a first set of coefficients for the second spatial audio processing and the second mode uses a second set of coefficients for the second spatial audio processing.

20. The method of claim 12, further comprising: At least one of the first spatial audio processing or the second spatial audio processing is adjusted based on a head orientation of a listener.

Images