Devices and methods for binaural audio rendering

Abstract

An audio processing apparatus (110, 120) for processing an input audio signal for a user (115) in an acoustic environment is disclosed. The audio processing apparatus (110, 120) is configured to obtain a first and a second sound signal from a sound source within the acoustic environment captured by a first and a second microphone (116a,b) and to determine a plurality of acoustic parameters based on the first and second sound signal, wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters and a second subset of orientation-dependent acoustic parameters. Moreover, the audio processing apparatus (110, 120) is configured to determine a first subset of a plurality of binaural room impulse responses, BRIRs, based on the first subset of environment-dependent acoustic parameters and to determine based on the first subset of the plurality of BRIRs a best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters. The audio processing apparatus (110, 120) is further configured generate a first and second driving signal for a first and second transducer (114a,b) based on the best-fitting BRIR and the input audio signal for rendering the input audio signal by the first and the second transducer (114a,b).

Description

[0001] Devices and methods for binaural audio rendering

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to audio processing and audio rendering in general. More specifically, the disclosure relates to devices and methods for binaural audio rendering.

[0004] BACKGROUND

[0005] Impulse responses in the time domain and transfer functions in the frequency domain can be used for rendering an audio signal, such as a speech signal, at a virtual source position. For many applications, in particular augmented reality (AR) applications, such as a virtual meeting, it is desirable that the simulated virtual environment is highly consistent with the acoustics of an actual real room. This is also known as plausibility of acoustic perception or room acoustic coherence.

[0006] WO2017 / 185663 discloses using the microphone and the loudspeaker of a smart device (e.g., VR headset, smartphone, and the like) to measure a room impulse response (RIR) in an actual room, and then to combine the measured RIR with a pre-selected head-related transfer function (HRTF) to render 3D audio.

[0007] Instead of directly using a measured RIR, US20170223478A1 discloses an approach for extending 3D audio rendering to match local environment acoustics by using static acoustic room parameters. A reverberation fingerprint of an actual room (defined by the volume and the frequency dependent reverberation time of the actual room) is measured and compared with the reverberation fingerprint of a reference room (already pre-measured). After this comparison, a binaural room impulse response (BRIR) of the actual room is modified based on a reference BRIR and the respective reverberation fingerprint of the actual room and the reference room.

[0008] US20210231488A1 discloses a method for synthesizing a BRIR based on a recorded speech signal. This is done by estimating the reverberation time acoustic parameter RT60 blindly from a speech or music signal based on the maximum likelihood method / distribution of signal decay rates. The reverberation time is determined up to 8 kHz and then extended to higher frequencies.

[0009] A data augmentation approach for expanding a RIR dataset is disclosed in Bryan, Nicholas, "Impulse Response Data Augmentation and Deep Neural Networks for Blind Room Acoustic Parameter Estimation", ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020. A dataset of 16 RIRs is extended by generating different reverberation tail lengths and different DRR (Direct-to-Reverberant Ratio) gains resulting in an extended dataset of 8000 RIRs.

[0010] SUMMARY

[0011] It is an objective to provide improved devices and methods for using BRIRs for immersive binaural audio rendering.

[0012] The foregoing and other objectives are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0013] According to a first aspect an audio processing apparatus is provided for processing an input audio signal for a user in a real acoustic environment, in particular a real room the user is located in. The audio processing apparatus is configured to obtain a first and a second sound signal from a sound source within the acoustic environment captured by a first and a second microphone. The first and the second microphone may be located at the left and right ear of the user and the first and second sound signal may be room impulse responses, e.g. impulse responses of the room the user is located in to an excitation signal, such as a clap, human speech or the like.

[0014] The audio processing apparatus is further configured to determine a plurality of acoustic parameters based on the first and second sound signal. The plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters characteristic of the acoustic environment, in particular the room, and a second subset of orientation-dependent acoustic parameters, wherein the orientation-dependent acoustic parameters characterize a relative orientation between the sound source and the first and second microphone. Thus, as used herein, an orientation dependent acoustic parameter may depend on orientation and position.

[0015] Moreover, the audio processing apparatus in a first main processing stage is configured to determine a first subset of a plurality of binaural room impulse responses, BRIRs, based on the first subset of environment-dependent acoustic parameters. The plurality of BRIRs may be stored in a database and the first subset may be selected from the database. The audio processing apparatus may be configured to transform a BRIR to a binaural room transfer function, BRTF, in the frequency domain and vice versa. The audio processing apparatus is in a second main processing stage further configured to determine, e.g. select based on the first subset of the plurality of BRIRs a, e.g. one bestfitting BRIR based on the second subset of orientation-dependent acoustic parameters.

[0016] Moreover, the audio processing apparatus is configured to generate a first and second driving signal for a first and second transducer based on the best-fitting BRIR and the input audio signal for rendering the input audio signal by means of the first and second transducer.

[0017] Thus, an improved audio processing apparatus is provided using BRIRs for immersive binaural audio rendering.

[0018] In a further possible implementation form, each BRIR of the plurality of BRIRs is associated to a room and determining, e.g. selecting based on the first subset of the plurality of BRIRs the best-fitting BRIR comprises determining, e.g. selecting the best-fitting BRIR from all or a subset of a circular set of BRIRs associated to a room a BRIR of the first subset of the plurality of BRIRs is associated to, based on the second subset of orientation-dependent acoustic parameters. Selecting the best-fitting BRIR from all or a subset of a circular set of BRIRs of the plurality of BRIRs may comprise selecting the best-fitting BRIR from all BRIRs of the same room or just all BRIRs that are associated to the same room and position (e.g. a subset of the room, if there is more than one position in a room, or any other subset). Thus, the first main processing stage may comprise determining, e.g. selecting a subset of rooms based on the respective / associated first subset of best BRIRs, which are selected based on the environment-dependent acoustic parameters. Thus, by selecting a first subset of BRIRs each BRIR is associated to a room. The first main processing stage, which may be considered as a selection of rooms, may include pruning, e.g. selecting only one BRIR per room (which may be enough for determining best rooms) and discarding other BRIRs of the same room. Thus, the first subset of the plurality of BRIRs may comprise only BRIRs associated to different rooms.

[0019] In a further possible implementation form, the best-fitting BRIR is associated with a circular set of BRIRs and the audio processing apparatus is configured to generate the first and second driving signal for the first and second transducer by processing the input audio signal based on one of the BRIRs of the circular set of BRIRs. This BRIR from the circular set of BRIRs may be selected based on information about the orientation of the head of the user (while rendering), which may be provided by a head tracking device. In a further possible implementation form, the audio processing apparatus is a headset device and the headset device comprises the first and second microphone configured to capture the first and second sound signal from the sound source and the first and second transducer are configured to be driven by the first and second driving signal.

[0020] In a further possible implementation form, the headset device is a headset device with active noise cancellation, ANC, and the first and second microphones are a first and second ANC microphone.

[0021] In a further possible implementation form, the audio processing apparatus is a mobile communication device, in particular a smartphone, a tablet computer or the like, wherein the mobile communication device comprises a communication interface configured to receive the first and second sound signal from the first and second microphone and to transmit the first and second driving signal to the first and second transducer for rendering the input audio signal based on the best-fitting BRIR via the first and second transducer.

[0022] In a further possible implementation form, the first subset of environment-dependent acoustic parameters comprises one or more monaural acoustic parameters.

[0023] In a further possible implementation form, the first subset of environment-dependent acoustic parameters comprises an inter-aural cross-correlation, ICC, parameter and / or one or more of the following monaural parameters: an energy decay curve, EDC, parameter; a reverberation time parameter, e.g. T / Teo; a definition parameter, e.g. D50; a clarity index parameter, e.g.

[0024] C50; a direct-to-reverberant ratio, DRR, parameter; a centre time parameter, e.g. Ts; and / or a perceptual mixing time parameter, e.g. tmp.

[0025] In a further possible implementation form, the audio processing apparatus is configured to select the first subset of the plurality of BRI Rs based on the first subset of environmentdependent acoustic parameters using a linear discriminant analysis, LDA, scheme.

[0026] In a further possible implementation form, the second subset of orientation-dependent acoustic parameters comprises one or more binaural acoustic parameters.

[0027] In a further possible implementation form, the second subset of orientation-dependent acoustic parameters comprises one or more of the following binaural acoustic parameters: an inter-aural level difference, ILD, parameter; an inter-aural time difference, ITD, parameter; and / or an inter-aural cross-correlation, ICC, parameter. In addition to these binaural parameters the audio processing apparatus may also use one or more monaural parameters in the second main processing stage.

[0028] In a further possible implementation form, the audio processing apparatus is configured to select the best-fitting BRIR based on the first subset of the plurality of BRI Rs based on the second subset of orientation-dependent acoustic parameters using a principal component analysis, PCA, scheme.

[0029] In a further possible implementation form, the audio processing apparatus is configured to send information about, e.g. an identifier of the best-fitting BRIR to a data processing apparatus, in particular cloud server and to receive, in response, the best-fitting BRIR or the circular set of BRI Rs associated with the best-fitting BRIR for generating the first and second driving signal for the first and second transducer.

[0030] According to a second aspect a method for processing an input audio signal for a user in an acoustic environment is provided. The method comprises the steps of: obtaining a first and a second sound signal from a sound source within the acoustic environment captured by a first and a second microphone; determining a plurality of acoustic parameters based on the first and second sound signal, wherein the plurality of acoustic parameters comprises a first subset of environmentdependent acoustic parameters characteristic of the acoustic environment and a second subset of orientation-dependent acoustic parameters, wherein the orientation-dependent acoustic parameters characterize a relative orientation between the sound source and the first and second microphone; determining, e.g. selecting a first subset of a plurality of binaural room impulse responses, BRI Rs, based on the first subset of environment-dependent acoustic parameters; determining, e.g. selecting based on the first subset of the plurality of BRI Rs a best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters; and generating a first and second driving signal for a first and second transducer based on the best-fitting BRIR and the input audio signal for rendering the input audio signal.

[0031] The method according to the second aspect can be performed by the audio processing apparatus according to the first aspect. Thus, further features of the method according to the second aspect result directly from the functionality of the audio processing apparatus according to the first aspect as well as its different implementation forms and embodiments described above and below. According to a third aspect a data processing apparatus, in particular a cloud server, is provided, wherein the data processing apparatus comprises a memory configured to implement a database for storing a plurality of binaural room impulse responses, BRI Rs. Moreover, the data processing apparatus comprises a communication interface configured to receive a plurality of acoustic parameters from an audio processing apparatus, wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters characteristic of an acoustic environment of the audio processing apparatus and a second subset of orientation-dependent acoustic parameters, wherein the orientationdependent acoustic parameters characterize a relative orientation between the sound source and the first and second microphone. The data processing apparatus further comprises a processing circuitry configured to determine, e.g. select a first subset of the plurality of BRIRs based on the first subset of environment-dependent acoustic parameters, wherein the processing circuitry is further configured to determine, e.g. select based on the first subset of the plurality of BRIRs a best-fitting BRIR based on the second subset of orientationdependent acoustic parameters. The communication interface is further configured to send information about the best-fitting BRIR to the audio processing apparatus.

[0032] In a further possible implementation form, the processing circuitry is configured to increase a number of the plurality of BRIRs stored in the memory by: modifying, e.g. equalizing a late reverberation tail of one or more BRIRs of the plurality of BRIRs; modifying, e.g. equalizing one or more sub-bands of a late reverberation tail of one or more BRI Rs of the plurality of BRI Rs; modifying one or more decay rates of one or more BRIRs of the plurality of BRIR; modifying one or more direct sound components of one or more BRIRs of the plurality of BRIRs; and / or adding one or more noise components to one or more BRIRs of the plurality of BRIRs.

[0033] According to a fourth aspect a method of operating a data processing apparatus, in particular a cloud server is provided. The method comprises the steps of: storing a plurality of binaural room impulse responses, BRIRs, in a memory of the data processing apparatus; receiving a plurality of acoustic parameters from an audio processing apparatus, wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters characteristic of an acoustic environment of the audio processing apparatus and a second subset of orientation-dependent acoustic parameters, wherein the orientation- dependent acoustic parameters characterize a relative orientation between the sound source and the first and second microphone; determining, e.g. selecting a first subset of the plurality of BRI Rs based on the first subset of environment-dependent acoustic parameters; determining, e.g. selecting based on the first subset of the plurality of BRI Rs a best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters; and sending information about the best-fitting BRIR to the audio processing apparatus.

[0034] The method according to the fourth aspect can be performed by the data processing apparatus according to the third aspect. Thus, further features of the method according to the fourth aspect result directly from the functionality of the data processing apparatus according to the third aspect as well as its different implementation forms and embodiments described above and below.

[0035] Embodiments of the invention can be implemented in hardware, in software or in any combination thereof.

[0036] According to a fifth aspect a computer program product is provided, comprising a computer- readable storage medium for storing program code which causes a computer or a processor to perform the method according to the second aspect or the method according to the fourth aspect, when the program code is executed by the computer or the processor.

[0037] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0038] BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In the following, embodiments of the present disclosure are described in more detail with reference to the attached figures and drawings, in which:

[0040] Fig. 1 is a schematic diagram illustrating an audio rendering apparatus, an audio processing apparatus and a data processing apparatus according to an embodiment for rendering an audio signal using BRIRs;

[0041] Fig. 2a is a diagram illustrating an exemplary BRIR according to an embodiment;

[0042] Fig. 2b is a diagram illustrating an exemplary BRIR with a left and right ear channel as well as some acoustic parameters according to an embodiment; Fig. 3a is a schematic diagram illustrating different exemplary spatial arrangements for generating a database of BRI Rs used by the audio processing apparatus and / or the data processing apparatus according to an embodiment;

[0043] Fig. 3b is a schematic diagram illustrating different exemplary spatial arrangements for using a database of BRIRs by the audio processing apparatus and / or the data processing apparatus according to an embodiment for rendering an input audio signal based on a bestfitting BRIR;

[0044] Fig. 4 is a schematic diagram illustrating processing stages implemented by an audio processing apparatus according to an embodiment for determining a best-fitting BRIR from a database of BRIRs;

[0045] Fig. 5a is a schematic diagram illustrating an exemplary spatial arrangement for rendering an audio signal based on a best-fitting BRIR from a database of BRIRs according to an embodiment and an additional source using an offset angle;

[0046] Fig. 5b is a schematic diagram illustrating the rendering of an audio signal by an audio rendering apparatus based on a best-fitting BRIR determined by an audio processing apparatus according to an embodiment;

[0047] Fig. 6 is a schematic diagram illustrating processing stages implemented by a data processing apparatus according to an embodiment for enlarging a database of BRIRs; Figs. 7 to 9 are schematic diagrams illustrating further details of the processing stages of figure 6 implemented by a data processing apparatus according to an embodiment for enlarging a database of BRIRs;

[0048] Fig. 10 is a diagram illustrating a visualization of subsets of a plurality of BRIRs based on linear discriminant analysis (LDA);

[0049] Fig. 11 is a schematic diagram illustrating pre-processing stages implemented by an audio processing apparatus according to an embodiment for determining a best-fitting BRIR from a database of BRIRs;

[0050] Fig. 12 is a diagram illustrating a visualization of a best-fitting BRIR of a plurality of BRIRs based on a principal component analysis (PCA);

[0051] Fig. 13 is a schematic diagram illustrating processing stages implemented by an audio processing apparatus according to an embodiment for determining a best-fitting BRIR from a database of BRIRs;

[0052] Fig. 14 is a flow diagram illustrating a method for determining a best-fitting BRIR from a database of BRIRs according to an embodiment; and

[0053] Fig. 15 is a flow diagram illustrating a method of operating a data processing apparatus for determining a best-fitting BRIR from a database of BRIRs according to an embodiment. In the following, identical reference signs refer to identical or at least functionally equivalent features.

[0054] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0056] For instance, it is to be understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically noted otherwise.

[0057] Figure 1 is a schematic diagram illustrating an audio rendering apparatus 110, an audio processing apparatus 120 and a data processing apparatus 130 according to an embodiment. As illustrated in the embodiment shown in figure 1 , the audio rendering apparatus 110 may be a headset or headphones 110, in particular an active noise cancellation, ANC, headset 110 worn or wearable by a user 115 (also referred to as listener 115), comprising a first left-ear transducer, e.g. loudspeaker 114a and a first left-ear microphone 116a as well as a second right-ear transducer, e.g. loudspeaker 114b and a second right-ear microphone 116b. The first left-ear transducer, e.g. loudspeaker 114a and the second right-ear transducer, e.g. loudspeaker 114b are configured to be driven by driving signals to generate corresponding sound signals for the ears of the user 115. The first leftear microphone 116a and the second right-ear microphone 116b are configured to capture a first and a second sound signal. In an embodiment, the ANC headset 110 may comprise ear cups, wherein the loudspeakers 114a,b are integrated in the ear cups and the microphones 116a,b are configured to capture the external sound outside of the ear cups. In further embodiments, the ANC headset 110 may be an in-ear, on-ear, or over-ear headset 110.

[0058] As further illustrated in figure 1 , the audio processing apparatus 120 may be a mobile phone, smartphone, tablet computer 120 or a similar mobile communication device 120 of the user 115. The audio processing apparatus 120 may comprise a processing circuitry 121, e.g. one or more processors 121 configured to process data for providing the functionality described in more detail further below. The audio processing apparatus may be connected to the audio rendering apparatus 110, e.g. via communication interfaces 113 / 123, to obtain the sound signals from and provide the driving signals to the audio rendering apparatus 110. The processing circuitry 121 may be implemented in hardware and / or software and may comprise digital circuitry, or both analog and digital circuitry. Digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. As illustrated in figure 1, the audio processing apparatus 120 may further comprise a communication interface 123 configured to communicate with the audio rendering apparatus 110 (for instance a Bluetooth interface or wire 123) and / or the data processing apparatus 130, in particular cloud server 130 (for instance a Wi-Fi or cellular communication network interface 123 for communicating via a cellular communication network 125 with the data processing apparatus 130). The audio processing apparatus 120 may further comprise a memory 125 configured to store a database of binaural room impulse responses, BRIRs, as will be described in more detail below, as well as executable program code which, when executed by the processing circuitry 121 , causes the audio processing apparatus 120 to perform the functions and methods described herein. Similar to the audio processing apparatus 120 the audio rendering apparatus 110 may comprise one or more processors 111, a communication interface 113 (in particular a Bluetooth communication interface 113) for communication with the audio processing apparatus 120 and a memory 115 as well.

[0059] As illustrated in figure 1 , the data processing apparatus 130 may be a cloud server 130. The data processing apparatus, in particular cloud server 130 may comprise processing circuitry 131 , e.g. one or more processors 131 configured to process data for providing the functionality described in more detail further below. The processing circuitry 131 may be implemented in hardware and / or software and may comprise digital circuitry, or both analog and digital circuitry. Digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. As illustrated in figure 1 , the data processing apparatus 130 may further comprise a communication interface 133 configured to communicate with the audio processing apparatus 120, for instance, via a Wi-Fi or cellular communication network 125. The data processing apparatus, in particular cloud server 130 may further comprise a memory 135 configured to store a database of BRI Rs, as will be described in more detail below, as well as executable program code which, when executed by the processing circuitry 131 , causes the data processing apparatus, in particular cloud server 130 to perform the functions and methods described herein.

[0060] For a better understanding of the embodiments some general information concerning audio processing and rendering using impulse responses will be introduced in the following.

[0061] As will be appreciated, a binaural room impulse response (BRIR) describes the acoustic relationship of a sound source and a listener in any given environment, for instance, a specific room and hence may be used to create an immersive 3D audio signal over headphones, such as the audio rendering apparatus 110 shown in figure 1. A BRIR is a special kind of impulse response (IR) (the time-domain equivalent of a transfer function, which is referred to as binaural room transfer function (BRTF)). As a room impulse response (RIR) it describes acoustic properties of a room, more specifically of a source-listener relationship in a room.

[0062] As will be further appreciated, a BRIR usually may comprise two channels of room impulse responses (RIRs), namely a left ear RIR and a right ear RIR. An example for the two channels of a BRIR is shown in figure 2b, which illustrates that a left ear RIR and right ear RIR might have a rather similar shape. An exemplary RIR is illustrated in figure 2a, which can be divided into three main temporal portions, namely a "direct sound" portion, an "early reflections" portion and a "late reverberation" portion (also referred to as "later tail"). The "direct sound" portion is usually defined by the shortest distance between the sound source and the receiver and obeys the inverse square law of sound signal propagation. Consequently, the "direct sound" portion is generally the main contributor for localizing the sound source. The "early reflections" portion of a RIR comprises the first few orders of reflections from the walls and / or objects of a room contributing both to the localization of the sound source and the acoustic reflection characteristics of the room. The "late reverberation" portion of an RIR comprises the higher order reflections and typically reaches the listener with no or only very little directional information. Thus, the diffused sound of the "late reverberation" portion of an RIR is the main contributor for the room perception.

[0063] As will be further appreciated, a BRIR does not capture the sound in free-field but at the two ears of the listener, which is related to the head-related transfer function (or impulse response) of the listener. As a result, it not only captures room acoustic aspects but also aspects of binaural hearing and the mechanisms of human sound localization, like interaural- time and level differences.

[0064] Typically, a BRIR can be recorded using a dummy head. One measurement of a BRIR can describe how sound would travel through the environment (e.g. a room interior) from a sound source to each ear of the listener. Since it is very much dependent upon relative source and listener orientation (and position), it is preferable to measure at least a circular set (azimuthal) of 360 degrees at one position. This may be achieved by rotating the dummy head and measuring the BRIR for each orientation. A circular set of BRIRs may contain, for instance, 360 different BRIRs, namely one BRIR for each source-listener respective orientation in steps of. e.g. 1 degree (or any other step width or granularity, e.g. 5 degrees). Once the circular set of BRIRs has been measured, during rendering, an orientation respective BRIR can be convolved with a desired input audio signal to achieve a plausible virtual sound source in the current environment. According to an embodiment, this may be achieved using a head tracking device (as also illustrated in figure 5b) configured to track the orientation of the head of the user 115, which would then help to select a specific BRIR of a circular set of BRIRs associated with a best-fitting BRIR, which will be described in more detail in the following, to be used for the convolution with the desired input audio signal.

[0065] In practice, it is much more complex as the perception of a sound source is usually room and listener dependent. Consequently, measuring the BRIR for each user in every possible room is virtually impossible. For this reason, a set of synthetic BRIR with a generic Head-Related Transfer Function (HRTF) may be used to create a plausible (realistic) illusion of the sound source. The perceived auditory impression relies on the external sound stimuli but also depends on the expectations about the auditory event in the actual room due to different room acoustics. The perceived externalization and plausibility may be reduced by applying such synthesized BRIRs without considering the actual room acoustics for binaural rendering. Hence, it is important that the simulated virtual environment should be highly consistent with the acoustics of an actual real room, especially for augmented reality (AR) applications as they require room coherence for achieving high immersion. For instance, a virtual musician in the current environment should sound the same as the real musician in the current environment.

[0066] Instead of measuring circular sets of BRI Rs for an actual real room, acoustic parameters may be estimated for adapting virtual 3D audio to the room. Acoustic parameters may be divided into environment-dependent acoustic parameters characteristic of the acoustic environment, in particular the room (herein referred to as "monaural acoustic parameters") and orientation-dependent acoustic parameters (herein referred to as "binaural acoustic parameters"), wherein the orientation-dependent acoustic parameters characterize a relative orientation between the sound source and the listener. As used herein, an orientation dependent acoustic parameter may depend in addition to the orientation also on the position of the listener within the room.

[0067] Embodiments disclosed herein may make use of one or more of the following environmentdependent, e.g. monaural acoustic parameters.

[0068] Energy Decay Curve (EDO) Defines the energy decay behavior of an impulse in a room

[0069] Reverberation Time (T / Tso Time taken for energy to decay by ‘x’ dB (generally 60 dB)

[0070] Definition (D50): Ratio of energy components up to 50ms after the direct sound to the total energy of RIR

[0071] Clarity Index (C50): Logarithmic ratio between early sound energy (until 50ms) to later

[0072] Direct-to-Reverberant Ratio (DRRY Ratio of direct part to the rest of the reverberation (similar to C50)

[0073] Centre Time (Ts): Time at which half of the signal energy has reached the receiver

[0074] Perceptual Mixing Time (tmp): The time of transition from early reflection to reverberations, after which no more directional information can be perceived

[0075] Embodiments disclosed herein may make use of one or more of the following orientationdependent, e.g. binaural acoustic parameters. Inter-aural Level Difference (ILD): Sound pressure level difference of the direct sound at each ear

[0076] Inter-aural Time Difference (ILD): Time / Phase difference of the direct sound at each ear

[0077] Inter-aural Cross Correlation (ICC): Cross-correlation of the signals at each ear (characteristic of spatial differences)

[0078] According to embodiments disclosed herein these monaural and binaural acoustic parameters are used to find a best-fitting BRIR which would result in a plausible, i.e. immersive illusion of the sound source, in particular in augmented reality (AR) applications. As will be described in more detail in the following, this may be done by recording a sound signal with the pair of Active Noise Cancellation, ANC, microphones 116a, b of the audio rendering apparatus 110, which then can be used to extract, e.g. determine one or more monaural and one or more binaural acoustic parameters based on the recorded sound signal.

[0079] More specifically, the audio processing apparatus 120 is configured to obtain a first and a second sound signal from a sound source, such as a clap or human speech, within the acoustic environment, such as room 180 (see, for instance, figure 3b) the user 115 is located in, as captured by the first and second microphone 116a, b of the audio rendering apparatus 110. As already described above, the first and the second microphone 116a,b may be located at the left and right ear of the user 115 and the first and second sound signal may be room impulse responses excited by a clap, speech or another type of excitation sound signal within the room 180 the user 115 is located in. Although in the following an embodiment will be described in the context of figure 1, where most of the audio processing is done by the audio processing apparatus 120, while audio capturing and rendering is done by the audio rendering apparatus 110, in other embodiments the audio processing may be also performed by the audio rendering apparatus 110 in the form of "smart headphones". In other words, in other embodiments the audio rendering apparatus 110 may comprise and / or provide the functionality of the audio processing apparatus 120.

[0080] As will be described in more detail below, the audio processing apparatus 120 is further configured to determine a plurality of acoustic parameters based on the first and second sound signal. The plurality of acoustic parameters comprises a first subset of environmentdependent acoustic parameters characteristic of the acoustic environment, in particular the room 180 the user 115 is located in, and a second subset of orientation-dependent acoustic parameters, wherein the orientation-dependent acoustic parameters characterize a relative orientation between the sound source, e.g. the clap and the first and second microphone 116a, b. As used herein, an orientation dependent acoustic parameter may depend on orientation and position.

[0081] Moreover, the audio processing apparatus 120 in a first main processing stage is configured to determine a first subset of a plurality of binaural room impulse responses, BRI Rs, based on the first subset of environment-dependent acoustic parameters. The plurality of BRI Rs may be stored in a database and the first subset may be selected from the database. In an embodiment, the database of BRI Rs or at least information thereof may be stored in the memory 125 of the audio processing apparatus 120 and / or in the memory 135 of the data processing apparatus 130, e.g. cloud server 130. During processing the audio processing apparatus 120 may be configured to transform a BRIR to a binaural room transfer function, BRTF, in the frequency domain and vice versa.

[0082] The audio processing apparatus 120 is in a second main processing stage further configured to determine, e.g. select based on the first subset of the plurality of BRI Rs a, e.g. one bestfitting BRIR (defining or representing the best combination of room and orientation respectively angle) based on the second subset of orientation-dependent acoustic parameters.

[0083] Moreover, the audio processing apparatus 120, e.g. a smartphone or tablet, is configured to generate a first and second driving signal for the first and second transducer 114a,b of the audio rendering apparatus 110, e.g. the ANC headphones 110, based on the best-fitting BRIR and a desired input audio signal for rendering the desired input audio signal by means of the first and second transducer 114a, b.

[0084] As already described above, in an embodiment, the database of BRIRs may be actually stored only or also in the memory 135 of the data processing apparatus 130, e.g. cloud server 130. For such an embodiment, the communication interface 133 of the data processing apparatus 130 is configured to receive the plurality of acoustic parameters from the audio processing apparatus 120, wherein the plurality of acoustic parameters comprises the first subset of environment-dependent acoustic parameters characteristic of the acoustic environment of the audio processing apparatus 120 and the second subset of orientationdependent acoustic parameters, wherein the orientation-dependent acoustic parameters characterize a relative orientation between the sound source and the first and second microphone 116a,b. In this embodiment, the processing circuitry 131 of the data processing apparatus 130 is configured to determine, e.g. select the first subset of the plurality of BRI Rs based on the first subset of environment-dependent acoustic parameters, and to determine, e.g. select based on the first subset of the plurality of BRIRs the best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters. The communication interface 133 of the data processing apparatus 130 is further configured to send information about the best-fitting BRIR to the audio processing apparatus 120. This information may contain the best-fitting BRIR itself and / or information for reconstructing or identifying the best-fitting BRIR in the database of BRIRs stored in the memory 125 of the audio processing apparatus 120.

[0085] In a further embodiment, the audio processing apparatus 120 may be configured to send information about, e.g. an identifier of the best-fitting BRIR to the data processing apparatus, in particular cloud server 130 and to receive, in response, the best-fitting BRIR or the circular set of BRIRs associated with the best-fitting BRIR from the cloud server 130 for generating the first and second driving signal for the first and second transducer 114a,b.

[0086] More detailed embodiments of the audio processing apparatus, e.g. smartphone 120 and the data processing apparatus, e.g. cloud server 130 will be described in the following.

[0087] Figure 3a is a schematic diagram illustrating different exemplary spatial arrangements (e.g. in room R01 for positions P0 and P1 and orientations A0, Aa and Ap, “0”, “a” and “ representing different orientation angles) of a dummy head within a test room for generating a database of BRIRs used by the audio processing apparatus 120 and the data processing apparatus 130 according to an embodiment, while figure 3b is a schematic diagram illustrating different exemplary spatial arrangements for using the database of BRIRs by the audio processing apparatus 120 and the data processing apparatus 130 according to an embodiment for a use case, where the listener 115 is located in an actual room 180. More specifically, figure3a shows how the dataset of BRIRs may be generated by measuring BRIRs at different positions and different orientations in a room and how the resultant BRIRs may be saved using a naming system (shown on the left-hand side of figure 3a). In figure 3b it is illustrated how the best-matching BRIR (here BRIR_R01_P01_Aa) may be found using a clap made at position “X” relative to the user 115 within the room 180. In a first stage a first subset of BRIRs and the respective rooms (e.g. here room R01 among others) is determined based on environment-dependent parameters, and in a second stage the best matching BRIR is determined based on orientation-dependent acoustic parameters. The best matching BRIR, here BRIR_R01_P01_Aa, defines the final / best room, here R01, best orientation or angle, here a, in relation to the central or reference angle at 0° of the circular BRIR set, and optionally the best position, here P01. The offset a then may be used to determine the BRIR of the circular set of BRI Rs (of the best matching BRIR) to be applied for plausibly rendering the virtual acoustic source(s).

[0088] As already described above, in an embodiment, a two-stage process may be used for determining the best-fitting BRIR from an existing (possibly enlarged) database of BRIRs. In an embodiment, the first stage includes finding the correct room using only single channel (monaural) parameters such as RT, C50, and the like (possibly including ICC for better results, when the binaural information is present). As will be described in more detail below, to quickly find a closest room based on different parameters dimensionality reduction techniques may be used. For instance, by performing a linear discriminant analysis (LDA) on the room-related parameters such as octave-band RT60, octave-band C50 and ICC of early reflections, the rooms associated with the different BRIRs of the database of BRIRs can be clustered into different sets, which may then be used during run-time to find the best-fitting room. As will be described in more detail below, the second stage involves finding the accurate relative listener-source position / orientation within the best room(s) determined by means of the first stage. For this stage, in an embodiment, the binaural parameters may be used along with channel dependent (for each channel) C50, DRR, and Initial Time-Delay Gap (ITDG). For the best-fitting room candidate(s) determined in the first stage an optimal source-listener orientation (defined by the best-fitting BRIR) may be determined using principal component analysis (PCA).

[0089] Figure 4 is a schematic flow diagram illustrating processing stages implemented by the audio processing apparatus 120 according to an embodiment for determining the best-fitting BRIR from an existing database of BRIRs. The processing flow illustrated in figure 4 comprises two main processing stages or blocks for determining the best-fitting BRIR. Generally, the first stage is configured to determine, e.g. select BRIRs corresponding to different room candidates (a subset or pre-selection of at least one but potentially a plurality of rooms), while the second processing stage uses this information to find the best-fitting BRIR for the selected room(s), e.g. an optimal room and listener-source orientation (corresponding to a direction of arrival, DOA) from the selected room(s). In figure 4 these two main stages are represented by the "Room Selector" processing block 409 and the "DOA Selector" processing block 411.

[0090] During run-time in a processing stage 404 the monaural and the binaural parameters are extracted from the sound signal 402 captured by the first and second microphone 116a,b of the audio rendering apparatus 110. In an embodiment, one or more of the extracted monaural and one or more of the extracted binaural parameters may be chosen for generating a parameter set, which may be represented as a vector and performing a dot product with vectors based on a linear discriminant analysis (LDA) that may be generated in setting up after enhancing the dataset of BRIRs, as illustrated by the blocks 401, 403, 405 and 407 of figure 4 and as will be described in more detail further below. For the room selection in block 409 the processor 121 of the audio processing apparatus 120 may be configured to use a spatial distance, such as the Mahalanobis distance. As will be explained in more detail below, the second main stage, e.g. the "DOA Selector" 411 may involve determining a dot product between a vector defined by the extracted binaural acoustic parameters and vectors generated for the existing database of BRIRs based on a principal component analysis (PCA) for finding the best-fitting BRIR 413, e.g. the optimal DOA and position of the sound source in the room selected during the first stage.

[0091] In an embodiment, once the best-fitting BRIR 413 has been selected, the corresponding circular set of BRIRs (including the selected best-fitting BRIR) may be used for rendering the input audio signal by means of the first and second transducers 114a, b of the audio rendering apparatus, e.g. headphones 110 to evoke the desired spatial audio impression (see processing block 415 of figure 4). By way of example, the selected best-fitting BRIR 413 may be the BRIR called "BRIR_R11_P2_E0_A-45.wav", wherein "R11" stands for the bestfitting room "11" and "P2" stands for the best-fitting position "2" within the best-fitting room "11". The corresponding circular set of BRIRs may cover orientations for this position "2" within the best-fitting room "11" spanning 360 degrees (A0 to A360 or A180 to A-180, wherein "A" stands for azimuth angle). In this example, "A-45" stands for the orientation (which corresponds to a direction of arrival, DOA) of the reference sound signal of about 45 degrees towards the left side of the listener 115 and "E0" means the elevation of the source is similar to the elevation of the listener 115.

[0092] Since in many cases a rotation of the room may not lead to a perceptual difference (excluding extreme cases), an offset angle may be used for the azimuth (A) for placing the virtual source on the same radius. An exemplary application scenario may be rendering a surround sound setup. In this case the reference sound signal may be a simple clap. Using the offset angles of the known surround sound setup, the virtual sources may be rendered in any given environment, as illustrated in figure 5a. In an embodiment, for rendering multiple sources at the same radius, the same room 180 and the same position of the listener 115 may be used by "rotating" the whole room 180 for a further source-listener orientation. Hence, in an embodiment, for a first source the circular BRIR set for the best-fitting BRIR may be used, while for a second source a rotated version (rotated by a specific offset, e.g. offset a in figure 3b) of the same circular BRIR set may be used. This is schematically illustrated in figures 5a and 5b as well as figure 3b.

[0093] As already described above, in an embodiment, the database of BRIRs stored, for instance, by the data processing apparatus, e.g. cloud server 130 may be enlarged by synthesizing further BRIRs based on the existing BRIRs (e.g. the BRIR circular set 601 shown in figure 6) within the database, which may completely or at least partially may comprise actually measured BRIRs. This increases the number of BRIRs that may the best-fitting BRIR for an actual room 180. As the late tail of a BRIR usually does not consist of any directional information after the perceptual mixing time, in an embodiment, the processing circuitry 131 of the data processing apparatus 130 may be configured to modify the respective late tail of one or more existing BRIRs of the database for creating a larger number of potentially fitting rooms. One way to do this is by generating new rooms (with different tonal characteristics) from the available dataset, e.g. a new circular BRIR circular set 660a for the same positions may be created by the data processing apparatus 130 (block 610 of figure 6). Additionally, as the early reflections consist of direction dependent information, by modifying only the later tail of circular BRIR sets more rooms, e.g. by creating different combinations of the sub-band reverberation, different tonal characteristics (e.g. high frequencies absorbent room by reducing decay time of only high frequency bands) of the room may be achieved (blocks 620 of figure 6) for further enlarging the database of BRIRs by adding a new BRIR circular set 660b. Moreover, altering the later tail decay rates, also in sub-bands (block 630 of figure 6) may result in different reverberation patterns that may be used for further enlarging the database of BRIRs. Furthermore, by suppressing or increasing the DRR (block 640 of figure 6) the data processing apparatus 130 may simulate different distances in the room itself (this can also be performed in sub-band responses) for further enlarging the database of BRIRs by generating a new circular set 660c. Lastly, by adding ambient or Gaussian noise to the resultant dataset with different SNR levels could result in noisy BRIR sets which might be used to train Neural Networks for detecting a clap in noisy environment (blocks 650 and 660d of figure 6). These approaches for enlarging the database of BRIRs will be described in more detail in the following.

[0094] As already described more generally in the context of figure 6, in an embodiment, the data processing apparatus 130 is configured to enlarge the database of BRIRs by modifying the reverberation tails of the already existing BRIR circular set 601 in different sub-bands, as illustrated in more detail in figure 7. This may be achieved by the data processing apparatus 130 by first breaking down a BRIR 703 of the BRIR circular set 601 into 12 Mel-frequency sub-bands (see 705 of figure 7). The direct sound remains untouched while different gains are applied on reverberation part in the frequency domain (see 707 of figure 7). After the equalization is applied, the result is added again in time domain to get the final result, e.g. the new BRIR and / or the corresponding circular set of BRI Rs 660a (see 707 and 709 of figure 7). The combination of the equalization filter may vary in different frequency sub-bands but the same equalization filter may be used for the whole BRIR set to keep the consistency. This will result in new rooms with different tonal characteristics, as defined by the new BRIR and / or the corresponding circular set of BRIRs 660a.

[0095] As already described more generally in the context of figure 6, in an embodiment, the data processing apparatus 130 is configured to enlarge the database of BRIRs by changing the decay time of the late reverberation tail of the already existing BRIR circular set 601, which may take into account the behavior at different frequencies, as illustrated in more detail in figure 8. In an embodiment, the data processing apparatus 130 is configured to extract the Mel-Frequency sub band RT60 and the perceptual mixing time from the existing circular BRIR set 601 (see 803, 807 of figure 8). A late tail may be generated using normal Gaussian noise (see 811 of figure 8) and filtered through the M el-frequency filter bank resulting in 12 Mel-frequency sub-bands. Using the extracted sub-band RT60, each sub band may be multiplied with a suitable decay filter with different lengths (see 809 of figure 8) to generate rooms with longer reverberations but similar decay behavior in frequency domain as the real room. Then, the respective late reverberation sub-bands and early and direct sub-bands are joined using the extracted mixing time (see 817 of figure 8). Finally, the sub-bands are summed up to achieve the new BRIR and / or the corresponding circular set of BRIRs 660b. Again, the mixing time and reverberations may be kept the same for a set of BRIR to keep consistency.

[0096] In an embodiment, also illustrated in figure 8, the data processing apparatus may be further configured to generate a decay filter to describe the decay of the later tail in different frequencies. But many rooms might have more absorbent material than others. So instead of always using a linear decay rate for the noise, different types of decays for the rooms may be used, according to an embodiment. For example, a more absorbent room might have a logarithmic decay in the higher frequencies than a linear decay as it absorbs high frequencies more and another room might have more reflective walls which might result in a brighter room. In that case a linear decay rate for high frequencies and logarithmic for the mid frequencies may be used according to an embodiment. Rooms have different materials of walls and objects inside the room, so choosing a decay type allows generating new rooms. So, by changing the decay rates of different sub-bands (see 809 of figure 8) it is possible to create further rooms with different characteristics. As already described more generally in the context of figure 6, in an embodiment, the data processing apparatus 130 is configured to enlarge the database of BRI Rs by altering the DRR of the already existing BRIR circular set 601, as illustrated in more detail in figure 9. This allows creating more possible distances of a source inside a room. In an embodiment, this is not done in the sub-band domain, since the other dataset augmentation steps might already result in different sub-band DRRs because of a change in the late reverberations, as illustrated in figure 9. In an embodiment, the processing is done in the time domain, by splitting a BRIR of the circular BRIR set 601 into two parts, namely the direct part (see 905 in figure 9) and the reverberant part (see 907 in figure 9). Different DRR gains are applied to the direct part (see 903 and 905 in figure 9) such that the illusion of distance is created. Finally, both the parts are summed up to create the new BRIR and / or the corresponding circular set of BRI Rs 660c (see 909 and 911 in figure 9). DRR gains are again kept constant during the processing of one circular set.

[0097] As already described more generally in the context of figure 6, in an embodiment, the data processing apparatus 130 is configured to enlarge the database of BRI Rs by adding noise components with different signal-to-noise ratios (SNRs) based on a trained noise robust neural network to an existing BRIR circular set. The parameters of the original BRIR circular set may be used and noise may be added to the signal. Hence, when using a clap to determine the BRIR the neural network may detect the behavior of ambient noise and result in better predictions.

[0098] As already described above, after the database of BRI Rs has been enlarged by one or more of the approaches described above, the data processing apparatus 130 may be configured to extract all acoustic parameters from each BRIR and then perform dimensionality reduction techniques such that only the actual relevant acoustic parameters are taken in account for determining a best-fitting BRIR. As will be appreciated, this dimensionality reduction reduces the computational complexity resulting in real-time selection of the best-fitting BRIR, for instance, by an audio processing apparatus 120 with limited computational resources, such as a smartphone 120.

[0099] After extracting all the acoustic parameters from the BRIR dataset, the parameters may be separated based on what they characterize, namely into monaural and binaural acoustic parameters, as already described above in the context of determining the best-fitting BRIR. Monaural acoustic Parameters such as sub-band EDC, sub-band RT60, C50, Mixing Time and ICC may be categorized as good room descriptors, while the channel based binaural acoustic parameters DRR, ILD, ITD and ICC may be categorized as localizers, as they do not play an important role in selection of the room.

[0100] Preferably, acoustic parameters are selected which are correlating to the human perception of a room. It has been discovered that parameters which provide a frequency dependent description of the room acoustics, such as sub-band EDC, sub-band RT60, and the like, correlate very well with human perception. In the following some combinations of acoustic parameters used according to different embodiments are disclosed:

[0101] • Parameters = [Octave-band RT, Octave-band DRR, ICC] o Number of parameters = 10+10+1 =21 o Describes the decay behavior in different octave-band regions and overall correlation of sound at each ear.

[0102] • Parameters = [RT_1kHz, C50] o Number of parameters = 2 o Describes decay behavior only at 1kHz and early-to-late ratio explaining how reverberant the room is.

[0103] • Parameters = [RT_mid_frequencies_average, DRRL-DRRR, C50L- C50R, ITDGL, ITDGR CC] o Number of parameters = 5 o Describing decay behavior in mid frequency range, early-to late ratios and how far are the walls using ITDG.

[0104] • Parameters = [EDT, DRRL, DRRR, C50L, , C50R,] o Number of Parameters = 6 o Describes energy decay of only 10 dB and early-to-late ratios

[0105] • Parameters= [Down-sampled Sub-band EDC] o Number of parameters = depending upon down-sampling / sparsity o Describes the energy decay behavior in different frequency bands.

[0106] In an embodiment, the dimensions of the parameters may be further reduced by considering each one as an independent parameter. To this end, in an embodiment, the data processing apparatus 130 is configured to perform a linear discriminant analysis (LDA) on the dataset and reduce the dimensions down to 2 or 3. As illustrated in figure 10, the LDA clusters the measurements from one room as one set and, thus, may be exploited for faster computations. Instead of matching all the parameters. LDA reduces the problem to finding the best-fitting BRIR in a lower dimensional space with, for instance, 2 or 3 dimensions. Thus, in order to save memory space, the data processing apparatus 130 may store only 2 or 3 linear discriminant values of the acoustic parameters for each BRIR instead of all the acoustic parameters. In an embodiment, these acoustic parameters may be stored as the whole LDA matrix along with the linear discriminant vectors (which are used during run-time) (see 1113 in figure 11). This allows reducing the complexity during the run-time, because during run-time only the relevant acoustic parameters need to be extracted by the audio processing apparatus 120 from the measured signal and a dot product with the linear discriminant vectors has to be determined (see 1301 , 1303, 1305 and 1307 in figure 13). This will put the measurement in the same space as other measurements. Once all the measurements are in the same space, the closest set is chosen using a spatial distance, such as the Mahalanobis distance (see 1309 in figure 13). This allows finding the nearest neighboring set, which essentially is the room candidate. In an embodiment, to find an optimal solution, the set with its center closest to the measurement may be determined.

[0107] As already described above, once the candidates for the room have been selected, an optimal solution for the listener-source orientation may be determined. Within the candidates set of BRIRs corresponding to all different orientations, the one orientation may be searched which provides the optimal perception. Whereas the room selection (i.e. the first stage) is mostly driven by parameters describing room acoustic properties, selecting the best orientation (i.e. the second stage) uses different acoustic parameters, as already described above in the context of figure 4. In an embodiment, only parameters are considered during this stage which are related to the localization of the sound source, such as channel-based DRR, ILD, ITD and ICC. This is because binaural parameters, such as ILD, ITD, are encoding the localization aspects (related to orientation), as already described above. Additionally, ICC (Inter-aural correlation) relates to the similarity of left and right ear signals and thus relates to the different sound components and together with DRR helps to describe direct and reverberant components in the selection process. Similar to the room selection process described above, the data processing apparatus 130 may be configured to perform a principal component analysis (PCA) on these parameters for every BRIR so that the dimensions of the parameter space may be reduced, for instance, to only two dimensions (see 1111 in figure 11). Since, PCA proves to give better in-class separation, all the positions / orientation for every rooms along with their principal component vectors may be saved in separate files (see 1115 in figure 11). In run-time, the binaural parameters from the recorded sound signal may be extracted (see 1303 and 1307 from figure 13) and the parameters may be multiplied using a dot-product with the PC vectors (see 1311 in figure 13) to map the measurement in the same space, as illustrated in figure 12 (marked by the circle). The nearest neighbor results in the best-fitting BRIR with an optimal DOA In this case, a Euclidean distance may be used to find the nearest neighbor. The complete runtime setup according to an embodiment is illustrated in figure 13. Furthermore, it is possible that fewer parameters are used for DOA selection, for example using only ILD or only ITD might provide different results. However, more optimal results may be achieved using the set of parameters which includes all the binaural parameters (ILD, ITD and ICC).

[0108] Figure 14 is a flow diagram illustrating a method 1400 for processing an input audio signal for a user 115 in an acoustic environment, in particular a room. The method 1400 comprises a step 1401 of obtaining a first and a second sound signal from a sound source within the acoustic environment captured by the first and a second microphone 116a,b. Moreover, the method 1400 comprises a step 1403 of determining a plurality of acoustic parameters based on the first and second sound signal, wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters and a second subset of orientation-dependent acoustic parameters. The method 1400 further comprises a step 1405 of determining a first subset of a plurality of BRIRs based on the first subset of environmentdependent acoustic parameters. Moreover, the method 1400 comprises a step 1407 of determining based on the first subset of the plurality of BRIRs a best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters. The method 1400 further comprises a step 1409 of generating a first and second driving signal for the first and second transducer 114a, b based on the best-fitting BRIR and the input audio signal for rendering the input audio signal.

[0109] The method 1400 can be performed by the audio processing apparatus 120 according to an embodiment. Thus, further features of the method 1400 result directly from the functionality of the audio processing apparatus 120 as well as its different embodiments described above and below.

[0110] Figure 15 is a flow diagram illustrating a method 1500 for operating the data processing apparatus 130, e.g. a cloud server 130. The method 1500 comprises a step 1501 of storing a plurality of BRIRs in the memory 135 of the data processing apparatus 130. Moreover, the method 1500 comprises a step 1503 of receiving a plurality of acoustic parameters from the audio processing apparatus 120, wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters and a second subset of orientation-dependent acoustic parameters. The method 1500 further comprises a step 1505 of determining a first subset of the plurality of BRIRs based on the first subset of environment-dependent acoustic parameters. Moreover, the method 1500 comprises a step 1507 of determining based on the first subset of the plurality of BRIRs a best-fitting BRIR based on the second subset of orientation-dependent acoustic parameters. The method 1500 further comprises a step 1509 of sending information about the best-fitting BRIR to the audio processing apparatus 120.

[0111] The method 1500 can be performed by the data processing apparatus, e.g. a cloud server 130 according to an embodiment. Thus, further features of the method 1500 result directly from the functionality of the data processing apparatus 130 as well as its different embodiments described above and below.

[0112] Embodiments disclosed herein address, in particular the following two technical challenges. Firstly, as measuring a circular BRIR set for every possible room is close to impossible leading to the unavailability of a huge diverse dataset consisting of real room scenarios, embodiments disclosed herein allow enlarging a dataset of BRI Rs, which may have been measured in a real room using a dummy head, resulting in a set of thousands of rooms which are as plausible as real rooms. Secondly, embodiments disclosed herein allow selecting a best matching BRIR set based on a multi-parameter approach. Embodiments disclosed herein allow considering different frequency bands of the same parameter (e.g. RT60) as independent parameters or a combination of multiple parameters such as full-band RT60, DRR, C50, ILD, ITD, ICC and Mixing Time or even a combination of both.

[0113] With respect to the prior art mentioned in the background section embodiments disclosed herein provide the following technical differences and advantages.

[0114] In the approach suggested in WO2017 / 185663 the BRIR is synthesized based on the obtained RIR and HRIR, which would result in incorrect mapping of the early reflections as the RIR does not contain directional information. In contrast thereto, embodiments disclosed herein so to say "measures" a first rough BRIR and finds a high-quality circular set of BRI Rs from the database. For the synthesis portion, WO2017 / 185663 suggests truncating the Room Impulse Response based on the length of the pre-selected HRTF (the mixing time is fixed and not room related). According to embodiments disclosed herein, the mixing time is part of the actual room acoustic parameters and may be used in the data augmentation stage to diversify the existing BRIR data with different possibilities of late reverberations.

[0115] The measurement of the room volume and the frequency-based reverberation time suggested in US20170223478A1 may be difficult in common consumer scenarios, since most consumers may not have the required hardware or technical knowledge to measure the reverberation time. An additional device or software to measure the room volume is needed. In addition, a quiet environment and relatively good equipment are required to perform the measurement of the reverberation time. Contrary thereto, embodiments disclosed herein do not require a prior knowledge of the room geometry and the reverberation time. Moreover, as US20170223478A1 only considers the reverberation time, it is possible that the virtual sound does not seem plausible due to a different reflection pattern. The source might be perceived closer or farther away as DRR is not considered. The location of the sound source (binaural parameters) is also not considered. Overall, ignorance of other parameters could result in an incorrect choice of a best-fitting BRIR.

[0116] The method disclosed in US20210231488A1 for synthesizing a BRIR based on a recorded speech signal also only considers one acoustic parameter, namely RT60 resulting in similar issues as mentioned above with respect to US20170223478A1. Moreover US20210231488A1 suggest synthesizing the late reverberation based on the current speech signal and further extending it to higher frequencies. This may result in incorrect high frequencies synthesis and an increased run-time computational complexity. Contrary thereto, the BRIR data enlargement approach implemented by embodiments disclosed herein may lead to a huge number of rooms considering all decay rates and different positions in those rooms and, consequently, improved results for high frequency synthesis. Furthermore, according to embodiments disclosed herein, dimensionality reduction may be used to reduce runtime complexity even though more than one acoustic parameter is used for selecting the best-fitting BRIR.

[0117] Embodiments disclosed herein may be used in numerous types of audio applications, where, for instance, a virtual sound source should be rendered in a room the listener is currently located in. For example, embodiments disclosed herein may be beneficially used in augmented reality applications to provide more coherent audio-visual acoustics, such as virtual assistance, virtual musicians or virtual surround sound setup. Another possible application of the embodiments disclosed herein are virtual conference systems, where several participants may be taking at the same time (so that the respective speech signals are coming from different directions) and each participant may select the participant he currently wants to listen to addressing the unwanted phenomena commonly known as "cocktail-party effect".

[0118] The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the present disclosure (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step). In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners.

[0119] For example, the described embodiment of an apparatus is merely exemplary. For example, the unit division is merely logical function division and may be another division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0120] The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

[0121] In addition, functional units in the embodiments of the invention may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Claims

CLAIMS1. An audio processing apparatus (110, 120) for processing an input audio signal for a user (115) in an acoustic environment (180), wherein the audio processing apparatus (110, 120) is configured to: obtain a first and a second sound signal from a sound source within the acoustic environment (180) captured by a first and a second microphone (116a, b); determine a plurality of acoustic parameters based on the first and second sound signal, wherein the plurality of acoustic parameters comprises a first subset of environmentdependent acoustic parameters and a second subset of orientation-dependent acoustic parameters; determine a first subset of a plurality of binaural room impulse responses, BRI Rs, based on the first subset of environment-dependent acoustic parameters; determine based on the first subset of the plurality of BRIRs a best-fitting BRIR (413) based on the second subset of orientation-dependent acoustic parameters; and generate a first and second driving signal for a first and second transducer (116a,b) based on the best-fitting BRIR (413) and the input audio signal.

2. The audio processing apparatus (110, 120) of claim 1, wherein each BRIR of the plurality of BRIRs is associated to a room and wherein determining based on the first subset of the plurality of BRIRs the best-fitting BRIR (413) comprises selecting the best-fitting BRIR (413) from all or a subset of a circular set of BRIRs associated to a room a BRIR of the first subset of the plurality of BRIRs is associated to, based on the second subset of orientationdependent acoustic parameters.

3. The audio processing apparatus (110, 120) of claim 1 or 2, wherein the best-fitting BRIR (413) is associated with a circular set of BRIRs and wherein the audio processing apparatus (110, 120) is configured to generate the first and second driving signal for the first and second transducer (114a, b) by processing the input audio signal based on one of the BRIRs of the circular set of BRIRs.

4. The audio processing apparatus (110, 120) of any one of claims 1 to 3, wherein the audio processing apparatus (110, 120) is a headset device (110) and wherein the headset device (110) comprises the first and second microphone (116a, b) configured to capture the first and second sound signal from the sound source and the first and second transducer(114a, b) are configured to be driven by the first and second driving signal.

5. The audio processing apparatus (110) of claim 4, wherein the headset device (110) is a headset device (110) with active noise cancellation, ANC, and wherein the first and second microphones (116a, b) are a first and second ANC microphone (116a, b).

6. The audio processing apparatus (110, 120) of any one of claims 1 to 3, wherein the audio processing apparatus (120) is a mobile communication device (120), wherein the mobile communication device (120) comprises a communication interface (123) configured to receive the first and second sound signal captured by the first and second microphone(116a,b) and configured to transmit the first and second driving signal to the first and second transducer (114a, b).

7. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the first subset of environment-dependent acoustic parameters comprises one or more monaural acoustic parameters.

8. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the first subset of environment-dependent acoustic parameters comprises an inter- aural cross-correlation, ICC, parameter and / or one or more of the following monaural parameters: an energy decay curve, EDC, parameter; a reverberation time parameter; a definition parameter; a clarity index parameter; a direct-to-reverberant ratio, DRR, parameter; a centre time parameter; and / or a perceptual mixing time parameter.

9. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the audio processing apparatus (110, 120) is configured to determine the first subset of the plurality of BRI Rs based on the first subset of environment-dependent acoustic parameters using a linear discriminant analysis, LDA, scheme.

10. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the second subset of orientation-dependent acoustic parameters comprises one or more binaural acoustic parameters.11 . The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the second subset of orientation-dependent acoustic parameters comprises one or more of the following binaural acoustic parameters: an inter-aural level difference, ILD, parameter; an inter-aural time difference, ITD, parameter; and / or an inter-aural crosscorrelation, ICC, parameter.

12. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the audio processing apparatus (110, 120) is configured to determine the best-fitting BRIR (413) based on the first subset of the plurality of BRI Rs based on the second subset of orientation-dependent acoustic parameters using a principal component analysis, PCA, scheme.

13. The audio processing apparatus (110, 120) of any one of the preceding claims, wherein the audio processing apparatus (110, 120) is configured to send information about the best-fitting BRIR (413) over a communication network (125) to a data processing apparatus (130) and to receive, in response, the best-fitting BRIR (413) for generating the first and second driving signal for the first and second transducer (114a, b).

14. A method (1400) for processing an input audio signal for a user (115) in an acoustic environment, wherein the method (1400) comprises: obtaining (1401) a first and a second sound signal from a sound source within the acoustic environment captured by a first and a second microphone (116a, b); determining (1403) a plurality of acoustic parameters based on the first and second sound signal, wherein the plurality of acoustic parameters comprises a first subset of environmentdependent acoustic parameters and a second subset of orientation-dependent acoustic parameters; determining (1405) a first subset of a plurality of binaural room impulse responses, BRIRs, based on the first subset of environment-dependent acoustic parameters; determining (1407) based on the first subset of the plurality of BRIRs a best-fitting BRIR (413) based on the second subset of orientation-dependent acoustic parameters; and generating (1409) a first and second driving signal for a first and second transducer (114a, b) based on the best-fitting BRIR and the input audio signal.

15. A data processing apparatus (130), comprising: a memory (135) configured to store a plurality of binaural room impulse responses, BRI Rs; a communication interface (133) configured to receive a plurality of acoustic parameters from an audio processing apparatus (110, 120), wherein the plurality of acoustic parameters comprises a first subset of environment-dependent acoustic parameters and a second subset of orientation-dependent acoustic parameters; and a processing circuitry (131) configured to determine a first subset of the plurality of BRI Rs based on the first subset of environment-dependent acoustic parameters, wherein the processing circuitry (131) is further configured to determine based on the first subset of the plurality of BRI Rs a best-fitting BRIR (413) based on the second subset of orientationdependent acoustic parameters; wherein the communication interface (133) is further configured to send information about the best-fitting BRIR (413) to the audio processing apparatus (110, 120).

16. The data processing apparatus (130) of claim 15, wherein the processing circuitry (131) is configured to increase a number of the plurality of BRI Rs stored in the memory (135) by: modifying a late reverberation tail of one or more BRIRs of the plurality of BRIRs; modifying one or more sub-bands of a late reverberation tail of one or more BRIRs of the plurality of BRIRs; modifying one or more decay rates of one or more BRIRs of the plurality of BRIR; modifying one or more direct sound components of one or more BRIRs of the plurality of BRIRs; and / or adding one or more noise components to one or more BRIRs of the plurality of BRIRs.

17. A method (1500) of operating a data processing apparatus (130), comprising: storing (1501) a plurality of binaural room impulse responses, BRI Rs, in a memory (135) of the data processing apparatus (130); receiving (1503) a plurality of acoustic parameters from an audio processing apparatus (110, 120), wherein the plurality of acoustic parameters comprises a first subset of environmentdependent acoustic parameters and a second subset of orientation-dependent acoustic parameters; determining (1505) a first subset of the plurality of BRI Rs based on the first subset of environment-dependent acoustic parameters; determining (1507) based on the first subset of the plurality of BRI Rs a best-fitting BRIR (413) based on the second subset of orientation-dependent acoustic parameters; and sending (1509) information about the best-fitting BRIR to the audio processing apparatus (110, 120).

18. A computer program product comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method (1400) of claim 14 or the method (1500) of claim 17, when the program code is executed by the computer or the processor.

Images