Devices and methods for binaural audio rendering

Abstract

An audio processing apparatus (100) is disclosed for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones (120) worn by a listener. The audio processing apparatus is configured to encode the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signal comprises a subset of (NMAX+1)2 spherical harmonic channels and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener. Moreover, the audio processing apparatus (100) is configured to generate a left ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a left ear head-related transfer function, HRTF, and / or head-related impulse response, HRIR, and a right ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR.

Description

[0001] DEVICES AND METHODS FOR BINAURAL AUDIO RENDERING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to audio processing and audio rendering in general. More specifically, the disclosure relates to devices and methods for binaural audio rendering, in particular over headphones.

[0004] BACKGROUND

[0005] Several approaches are known for reproducing 3D audio over headphones. Usually, these approaches require the use of a head-related transfer function (HRTF) or the time equivalent head-related impulse response (HRIR) in the processing chain to generate binaural signals for the left and right ear to be delivered, for instance, via headphones. The HRTF describes the acoustical transmission path for a sound source positioned somewhere in space to the listener’s ears. Head-tracking is often employed to provide a continuous metadata stream of the listener’s head orientation (also referred to as 3DOF head-tracking, i.e. head-tracking with 3 degrees of freedom). The binaural audio may then be adapted to ensure sources stay in the correct position when the head of the listener rotates. This creates dynamic cues which are crucial for resolving front-back ambiguities. If head-tracking is not employed, when the listener rotates, then the sound scene will remain fixed relative to the listener and rotate with them (a classic issue of standard stereo headphone playback). Head-tracking with 6 degrees of freedom (referred to as 6DOF head-tracking) includes further positional data of the listener. With this information, virtual sources may not just be given a direction of arrival, but also a physical position that the listener may then move around.

[0006] Binaural rendering approaches might be split into two categories, namely object-based reproduction and scene-based reproduction. In object-based reproduction each sound source is treated as an individual object and metadata specifying attributes about the sound source, such as azimuth and elevation angles of arrival, are encoded along with the mono audio stream. The number of input audio signals is equal to the number of objects, and thus rendering costs grow as this number increases. The approach to render each object to binaural signals is then repeated for every object individually, before summing all the left and right ear signals respectively to create a single binaural stream. In scene-based reproduction individual sound sources are first encoded and mixed into a single scene representation. Alternatively, a recording from a microphone array might capture the whole soundfield scene including diffuse sounds. From this stage on the number of audio channels is fixed, regardless of increasing or decreasing the number of objects. The encoding often involves representing the whole soundfield via a relevant mathematical basis, thus the scene is decomposed onto the basis and becomes an approximate representation of the original soundfield. Once in this format general manipulations to the whole soundfield may be applied easily. It is difficult, however, to extract or edit any given individual object again. Finally, the scene is rendered to binaural either using a virtual loudspeaker approach or by convolution with a HRTF represented in the relevant basis. Popular forms of scene-based audio include classic stereo recording approaches and Higher Order Ambisonics (HO A).

[0007] Channel-based audio is another crucial audio format, which, however, does not naturally lend itself to binaural headphone-based reproduction. Channel-based audio defines the loudspeaker signals for a specific loudspeaker layout, for example a stereo loudspeaker pair or a 5.1 loudspeaker array. Therefore, object-based approaches are generally employed to auralise such audio content. Here each loudspeaker may be modelled as a stationary object. Channelbased audio remains a standard across the whole audio industry. Common formats include stereo, 5.1, 5.1.2 and 7.1.

[0008] SUMMARY

[0009] It is an objective to provide improved devices and methods for computationally efficient, yet accurate binaural audio rendering, in particular over headphones.

[0010] The foregoing and other objectives are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0011] According to a first aspect an audio processing apparatus is provided for generating binaural audio rendering signals for audio rendering one or more audio input signals, which may define a soundfield, over headphones worn by a listener. The audio processing apparatus according to the first aspect is configured to encode the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signal comprises a subset of (NMAX+1)2spherical harmonic channels (herein also referred to as a plurality of spherical harmonic channels) and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener. The reference axis of the spherical harmonic representation may be the z axis of the reference system of the spherical harmonic representation.

[0012] Moreover, the audio processing apparatus according to the first aspect is configured to generate a left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a left ear head-related transfer function, HRTF, and / or head-related impulse response, HRIR, and to generate a right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR. By using a suitable subset of the plurality of spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals the audio processing apparatus according to the first aspect allows generating the binaural audio rendering signals in a computationally efficient manner in accordance with a desired accuracy.

[0013] In a further possible implementation form, the audio processing apparatus according to the first aspect is further configured to dynamically select the subset of the (NMAX+1)2spherical harmonic channels. This allows selecting the most suitable spherical harmonic channels of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals as the subset. In a further possible implementation form, the subset of the (NMAX+1)2spherical harmonic channels may be predefined or fixed.

[0014] In a further possible implementation form, the audio processing apparatus according to the first aspect comprises a user interface, wherein the user interface is configured to receive a user input and wherein the audio processing apparatus is configured to select the subset of the (NMAX+1)2spherical harmonic channels based on the user input. In a further possible implementation form, the audio processing apparatus is configured to obtain or estimate an energy associated with each of a plurality of spherical harmonic channels of a spherical harmonic representation of the left ear HRTF and / or HRIR and an energy associated with each of a plurality of spherical harmonic channels of a spherical harmonic representation of the right ear HRTF and / or HRIR and to select the subset of the plurality of, i.e. the (NMAx+l)2spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals based on the energy of each spherical harmonic channel of the spherical harmonic representation of the left ear HRTF and / or HRIR and the energy of each spherical harmonic channel of the spherical harmonic representation of the right ear HRTF and / or HRIR.

[0015] In a further possible implementation form, the audio processing apparatus is configured to select the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals based on the degree (often referred to as the degree m) of the spherical harmonic coefficients of the spherical harmonic representation of the left ear HRTF and / or HRIR and / or the degree m of the spherical harmonic coefficients of the spherical harmonic representation of the right ear HRTF and / or HRIR, wherein the reference axis, i.e. z axis of the spherical harmonic representation coincides with the listener’s interaural axis.

[0016] In a further possible implementation form, for encoding the one or more input audio signals into the spherical harmonic representation based on the plurality of spherical harmonics from order 0 to order NMAX and defined relative to the reference axis the audio processing apparatus is configured to: encode the one or more input audio signals into a preliminary spherical harmonic representation based on the plurality of spherical harmonics from order 0 to order NMAX and defined relative to a vertical axis, wherein the preliminary spherical harmonic representation of the one or more input audio signals comprises the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels and wherein the vertical axis of the preliminary spherical harmonic representation does not coincide with the interaural axis of the listener; and rotate the preliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals based on an orientation of interaural axis of the listener relative to the vertical axis. In a further possible implementation form, the reference axis of the spherical harmonic representation defines the z-axis of a reference system of the spherical harmonic representation and the interaural axis of the listener lies in the xy-plane of the reference system of the spherical harmonic representation, wherein for rotating the preliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals based on the orientation of interaural axis of the listener relative to the reference axis the audio processing apparatus is configured to: rotate the preliminary spherical harmonic representation of the one or more input audio signals into a further spherical harmonic representation of the one or more input audio signals so that the interaural axis of the listener coincides with the y-axis of the reference system of the spherical harmonic representation; and rotate the further preliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals so that the interaural axis of the listener coincides with the z-axis of the reference system of the spherical harmonic representation.

[0017] In a further possible implementation form, the number NCH of spherical harmonic channels of the subset of the plurality of, i.e. the (NMAx+l)2spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals is larger than NMAX+1 and smaller than (NMAX+1)2.

[0018] In a further possible implementation form, the audio processing apparatus, for frequencies below a frequency threshold value, is configured to generate the left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the left ear HRTF and / or HRIR and to generate the right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the right ear HRTF and / or HRIR. For frequencies above the frequency threshold value, the audio processing apparatus is configured to generate the left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the left ear HRTF and / or HRIR and to generate the right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the right ear HRTF and / or HRIR.

[0019] In a further possible implementation form, for generating the left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the zero-phase filter representation of the left ear HRTF and / or HRIR and for generating the right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the zero-phase filter representation of the right ear HRTF and / or HRIR the audio processing apparatus, for frequencies above the frequency threshold value, is configured to filter the left ear HRTF and / or HRIR and the right ear HRTF and / or HRIR with an all-pass filter adapted to the frequency threshold value.

[0020] According to a second aspect headphones are provided for audio rendering a left ear and right ear binaural audio rendering signal, wherein the headphones comprise an audio processing apparatus according to the first aspect.

[0021] According to a third aspect an audio processing method for generating binaural audio rendering signals for audio rendering one or more audio input signals, e.g. a soundfield, over headphones worn by a listener is provided. The audio processing method comprises the steps of encoding the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signal comprises a subset of (NMAX+1)2spherical harmonic channels and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener; and generating a left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a left ear head-related transfer function, HRTF, and / or head-related impulse response, HRIR, and a right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR. By using a suitable subset of the plurality of spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals the audio processing method according to the second aspect allows generating the binaural audio rendering signals in a computationally efficient manner in accordance with a desired accuracy.

[0022] The audio processing method according to the third aspect can be performed by the audio processing apparatus according to the first aspect. Thus, further features of the audio processing method according to the third aspect result directly from the functionality of the audio processing apparatus according to the first aspect as well as its different implementation forms and embodiments described above and below.

[0023] According to a fourth aspect a computer program product is provided, comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method according to the third aspect, when the program code is executed by the computer or the processor.

[0024] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In the following, embodiments of the present disclosure are described in more detail with reference to the attached figures and drawings, in which:

[0027] Fig. 1 is a schematic diagram illustrating an example of an audio processing apparatus implementing HRIR synthesis for a single audio object;

[0028] Fig. 2 is a schematic diagram illustrating an example of an audio processing apparatus implementing a virtual loudspeaker array combining object and channel-based input;

[0029] Fig. 3 is a schematic diagram illustrating an example of an audio processing apparatus implementing HO A rendering combining object and B-format input; Fig. 4 is a schematic diagram illustrating an example of an audio processing apparatus implementing HOS rendering combining object and B-format input;

[0030] Fig. 5 is a schematic diagram illustrating an audio processing apparatus according to an embodiment for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones;

[0031] Fig. 6 is a schematic diagram illustrating a rotation of the interaural axis of the listener relative to the reference system of a spherical harmonic representation implemented by an audio processing apparatus according to an embodiment;

[0032] Fig. 7a and 7b illustrate HRTF spherical harmonic coefficients before (figure 7a) and after (figure 7a) a rotation of the interaural axis of the listener relative to the reference system of a spherical harmonic representation implemented by an audio processing apparatus according to an embodiment;

[0033] Fig. 8 illustrates the selection of a subset of coefficients implemented by an audio processing apparatus according to an embodiment; and

[0034] Fig. 9 is a flow diagram illustrating steps of an audio processing method according to an embodiment for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones.

[0035] In the following, identical reference signs refer to identical or at least functionally equivalent features.

[0036] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0038] For instance, it is to be understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically noted otherwise.

[0039] Before describing detailed embodiments in the following some technical background as well as terminology will be introduced making use of one or more of the following abbreviations:

[0040] HOS Higher Order Stereo

[0041] MOS Mixed Order Stereo

[0042] MOS-Hp Mixed Order Stereo Over Headphones

[0043] HO A Higher Order Ambi sonics

[0044] HRTF Head-Related Transfer Function

[0045] HRIR Head-Related Impulse Response

[0046] 3DOF / 6DOF 3 / 6 Degrees of Freedom

[0047] As used herein, binaural audio is a two channel (left and right ears) 3D audio format created using HRTFs for playback over headphones. As used herein, HO A B-Format is a multichannel 3D audio format used to represent a soundfield using a spherical harmonic decomposition (i.e. representation).

[0048] As used herein, 3DOF / 6DOF describe degrees of freedom for listener tracking. 3DOF covers listener rotations, while 6DOF includes rotations and translations.

[0049] As used herein, the head-related transfer function (HRTF, frequency domain) or the equivalent dead-related impulse response (HRIR, time domain) describe the acoustical transmission path for a sound source positioned somewhere in space to the listener’s ears.

[0050] As used herein, MOS Format is a multichannel 3D audio format which is analogous to HOA B-format, except using the MOS approach.

[0051] The classical approach to render binaural audio is an object-based method where each individual sound source is directly convolved in real time with a relevant HRIR, to reproduce the illusion of that sound source at any given position around the listener. Head-tracking may be employed, in which case the positions of the sound sources are compensated for any head rotations and the correct HRIR is selected for each processing block.

[0052] The typical signal flow for this approach is illustrated in figure 1. A single mono audio object and its associated positional metadata is the input to the conventional audio processing apparatus 10 of figure 1, along with the listener head-tracking metadata. A metadata controller 11 combines the head-tracking and object metadata to identify the correct HRIR filters that may be selected from a HRIR dataset 12. The object audio input is then convolved with these filters by a HRIR convolution unit 13, resulting in the desired binaural output. This process must be repeated for each audio object individually.

[0053] HRIR rendering is advantageous as it reproduces the desired binaural signals correctly across the full frequency spectrum. Furthermore, it only contains a single processing operation on the audio stream for each object. However, as the number of convolutions required increases linearly with the number of object inputs, the computational cost can increase significantly when rendering complicated scenes. Furthermore, the approach is dependent on a well- defined HRIR dataset to perform well. This HRIR dataset should have significant measurement points across the whole sphere, to render sound sources at all positions around the listener. The required number of measurements can be in the order of 1000's of points. Interpolation techniques can be employed to cover regimes where no HRIR filters are defined, at a further computational cost. Finally, HRIR rendering cannot be used to render diffuse soundfields (for example, crowd noise, reverberation, ambient sound), only objects at defined positions in space.

[0054] Loudspeaker panning techniques remain a further popular approach in spatial audio, which have developed from standard stereo and surround sound channel-based audio to full sphere panning technique. To create binaural renderings, such approaches may be simulated using a virtual loudspeaker array. This approach is widely used, as it retains compatibility with existing audio content mixed in stereo / surround sound and may also be used to auralise other approaches, such as HOA.

[0055] An example of an audio processing apparatus 20 implementing virtual loudspeaker rendering is shown in figure 2. The positions and number of virtual loudspeakers are defined before runtime. The Tenderer 20 can accept mono audio objects or channel-based audio. To work with objects, a panning algorithm 25 is implemented by the Tenderer 20 to create the correct signals for each loudspeaker channel. Generally panning algorithms are forms of amplitude panning, and so applying the algorithm is a simple gain matrix operation, defined by the Panning Metadata Controller 21. The panning algorithm 25 combines both head-tracking and object metadata with the pre-defined loudspeaker positions to ensure the object is placed correctly around the listener. Multiple objects can be dealt with by repeating this step, then mixing the corresponding loudspeaker signals together.

[0056] Following this, pre-mixed channel-based audio might also be combined with the loudspeaker signals, if the channel-based audio matches the virtual loudspeaker layout. Finally, the loudspeaker signals are auralised by the Virtual Loudspeaker Convolution 23, which convolves the loudspeaker signals with the correct HRIR filters for each loudspeaker position as selected from a HRIR dataset 22 based on information provided by the Virtual Loudspeaker Controller 24. Head-tracking data might also be employed at this stage, resulting in the loudspeakers remaining in fixed positions about the listener. However, for some applications such as auralising HOA, head rotation compensation is applied elsewhere in the processing chain and therefore not required at this stage. Virtual loudspeaker rendering is a useful technique as it ensures compatibility with legacy channel-based content such as stereo and surround sound mixes. Furthermore, it can be used to reproduce audio objects with a relevant panning algorithm which is also computationally simple to apply. The computational load due to convolutions is also fixed by the number of loudspeakers in the virtual array, which can range from a stereo pair of 2 loudspeakers to a considerable number sampling a sphere around the listener. Regardless, the number of convolutions is independent of the number of objects to be reproduced unlike with dynamic HRIR rendering. A key disadvantage to the approach is that panning algorithms generally cannot represent a diffuse sound field, only an audio object. Furthermore, where headtracking is required to adapt the loudspeaker positions with head movements, an extremely large HRIR dataset is required to cover every possible head position for each individual loudspeaker, and often only yaw rotations are accounted for.

[0057] HOA is a full end-to-end theory covering the recording, transmission and reproduction of a soundfield. From an analytical point of view HOA relies on the spherical harmonic expansion (also referred to as spherical harmonic representation) of a soundfield about a central position. An exemplary HOA based binaural Tenderer 30 is shown in figure 3. The Tenderer 30 may accept object audio input, along with the associated object metadata. A simple operation performed by the HOA Object Encoder 34 consists of applying a set of gains as defined by the Encored Metadata Controller 31 on the basis of input Object Metadata, encodes the mono object into the (IV + I)2B-format signals. Multiple objects may be processed by repeating this step, changing the gains as per the object metadata and mixing each of the corresponding B-format channels together. At this point, other B-format signals may be mixed into the processing chain, for example those recorded from a microphone array. Next, alterations to the B-format signal may be applied. In the binaural Tenderer case, a rotation may be applied by the HOA Rotation block 35 to account for the head orientation of this listener, as provided by Head-tracking Data processed by the Rotation Metadata Controller 33. This process is also a simple gain matrix operation.

[0058] Finally, the B-format signals are auralised by the Binauralisation block 36 to create a binaural output. This may be performed in two key approaches, by filtering in the spherical harmonic domain or by using a virtual loudspeaker array. Under certain conditions, both approaches are equal. The virtual loudspeaker array approach requires a further gain matrix to define the (IV + I)2loudspeaker signals, which are then convolved with the corresponding HRIR filters for each loudspeaker position, loaded from the HRIR Dataset 32. The spherical harmonic domain approach requires first sampling the HRIR of the HRIR Dataset 32 across a sphere (IV + I)2times, then performing a spherical harmonic decomposition of the HRIR. This defines a set of filters to be convolved with each B-format signal to create a binaural output. Both approaches require (IV + I)2X 2 filters / convolutions. Many pre-processing techniques exist when defining the binauralisation to minimise the effects of spatial aliasing in orders above the truncation order.

[0059] HOA is advantageous over many spatial audio techniques in its natural ability to represent and reproduce soundfields of all forms, including diffuse and ambient sounds as well as distinct audio objects. Furthermore, it is compatible with channel-based audio by representing the corresponding loudspeakers as audio objects in the correct position. It is also unique in allowing for the manipulation of the soundfield through a variety of operations, such as rotations, which are often implemented through computationally simple gain matrix operations. The number of audio channels required can be significant, however is fixed by the truncation order of the approach to (A + I)2. For very complex scenes this can be advantageous over other techniques, however for simple scenes the number of channels can be comparatively large. Furthermore, the HRIR dataset requires very specific measurement points of the listener's head to ensure proper sampling across the sphere, and therefore optimal binaural reproduction. However, the number of HRTF measurement points is significantly less than for other techniques. Finally, the truncation order sets an upper frequency limit on the accurate reproduction of the soundfield, above which spatial aliasing occurs. For low orders this means the binaural signals are only accurate below a given frequency given by the N = kr rule of thumb, with k the wavenumber and r the radius of the head of the listener.

[0060] Higher-Order Stereophony (HOS) is an alternative approach that shares many similarities with HOA. The technique may be expressed using two different types of decomposition of the soundfield. Here the soundfield is a representation using an expansion over a mathematical basis. Both decompositions lead to similar mathematical results. The first decomposition is the use of the Taylor expansion in ID along the interaural axis of the listener, i.e. the line defined by the listener’s ears. The second decomposition is using the spherical harmonics as with HOA, however first rotating the coordinate system so the interaural axis, i.e. the listener’s ears lie along the Cartesian z axis and then using only the zero degree spherical harmonics. This means HOS, like HOA, can render object audio input, microphone array input or channel-based audio content. Both Taylor and spherical harmonics basis decomposition require an infinite number of terms for complete accuracy, therefore the technique is truncated to an order N, which limits the number of audio channels to a finite number but means the soundfield is only accurately rendered below a high frequency threshold. HOS follows the same N = kr rule of thumb as HOA defining this high frequency threshold. Above the high frequency threshold spatial aliasing occurs and the soundfield is no longer accurately reproduced. However, the HOS soundfield representations are more compact than HOA, which means less audio channels are required to represent the soundfield, only (N+l). This comes at the cost of a reduction in accuracy of the soundfield representation.

[0061] The fundamental difference between HOS and HOA, is that HOS reproduces the soundfield accurately along a single line (which the listener’s ears lie upon), whilst HOA reproduces the soundfield accurately across a region (in 2D a circle, in 3D a sphere). By reducing the reproduction region to just accurate soundfield reproduction over a line, HOS introduces some error into the final binaural signals compared to HOA, is, however, a more compact reproduction scheme, requiring significantly less audio channels in the rendering procedure. Therefore, HOS is more efficient but less accurate than HOA.

[0062] The HOS representation creates sparsity through the use of a rotation of the reference coordinate system. In standard spatial audio coordinate systems the interaural axis of the listener is assumed to lie along the cartesian y axis. A rotation of the head to align the interaural axis, i.e. the ears of the listener with the z axis utilises special properties of the spherical harmonic representation of the HRTF to reorder the energy of the HRTF into the zero degree spherical harmonic channels (which to truncation order N is only (N+l) audio channels). This operation is referred to as the HOS rotation. Removal of channels to keep only those with m=0 is referred to as the HOS signal decimation.

[0063] Figure 4 shows an example of an audio processing apparatus 40 implementing a HOS renderer signal chain. The Tenderer 40 may accept object audio input, along with the associated object metadata. The HOS Object Encoder 45 applies a set of gains and encodes the mono object into the (A + 1) HOS format signals. The gain values are provided by the Encoder Metadata Controller 41, which takes the Object Metadata and Head-tracking Data as input. Multiple objects may be processed by repeating this step, changing the gains as per the object metadata and mixing each of the corresponding HOS format channels together. At this point, HOA B-format signals may be mixed into the chain, for example those recorded from a microphone array, these signals must under the HOS rotation and signal decimation, performed by the HOS Rotation and Signal Decimation 46. In the binaural Tenderer case, a rotation is applied to account for the head orientation of this listener, based on data provided by the Rotation Metadata Controller 43. This process is also a simple gain matrix operation.

[0064] Finally, the B-format signals are auralised by the Binauralisation block 47 to create a binaural output. This may be performed in three key approaches, by filtering in the spherical harmonic domain, filtering in the Taylor domain or by using a virtual loudspeaker array. Under certain conditions, all approaches are equal. All approaches require (A + 1) x 2 filters / convolutions. Many pre-processing techniques exist when defining the binauralisation to minimise the effects of spatial aliasing in orders above the truncation order. This operation requires the HRIRs of the HRIR Dataset 42 to be processed (off-line or in real-time) by the HRIR HOS Rotation and Decimation block 44, to make the HRIR data compatible with the HOS format.

[0065] As already described above, both HOA and HOS can be used to render various signal inputs (audio objects, channel based audio, microphone array signals, existing HOA B-format) to binaural, including listener headtracking. This means HOA and HOS can handle a range of different soundfields (e.g. audio objects, diffuse soundfields, reverberation).

[0066] HOA represents the soundfield using a decomposition onto the spherical harmonic basis. Each spherical harmonic ends up being an individual audio channel. To Nth order reproduction, (A + I)2audio channels are required. Increasing the order A increases the accuracy of the representation to a high frequency threshold. When rendering HOA to binaural using a HRTF, each spherical harmonic channel also requires convolution with a filter defined using the HRTF. To create these filters, the HRTF must be measured for at least (A + I)2sound source positions around the head, which should also ideally be in specific positions around a 3D sphere. HOA is a scene-based format, therefore the number of audio channels, (A + I)2, is fixed regardless of the number of audio objects. For appropriate broadband reproduction across the human hearing frequency range, this leads to a large number of audio channels and a very large computation load. Furthermore, filter convolutions are computationally expensive and a large number are also required (one per channel per ear = 2 x (A + I)2). HOS represents the soundfield using a decomposition onto a basis defined through a Taylor expansion or equivalently through a spherical harmonic basis. Each basis function ends up being an individual audio channel. Importantly, HOS requires specific orientation of the reference coordinate system and the listeners head position to create a sparser representation of the soundfield. This means for Nth order reproduction, only (IV + 1) audio channels are required. Increasing the order N increases the accuracy of the representation to a higher frequency limit. When rendering HOS to binaural using a HRTF, each spherical harmonic channel also requires convolution with a filter defined using the HRTF. To create these filters, the HRTF must be measured for at least (IV + 1) sound source positions around the head. HOS is a scene-based format, therefore the number of audio channels, (IV + 1), is fixed regardless of the number of audio objects.

[0067] Often devices implementing an audio processing apparatus for rendering 3D audio have limited computational resources, such as, for instance, mobile devices. The computational load of a Tenderer increases with the number of operations in the signal chain and the number of channels required. Embodiments disclosed herein provide audio processing apparatuses and methods for rendering 3D audio over headphones offering a flexible choice between computational load and accuracy, by choosing which audio channels can be rendered. As will be described in more detail in the following, embodiments disclosed herein provide audio processing apparatuses and methods offering a choice between the two extremes and individual benefits of HOA (many channels, highest accuracy) and HOS (fewest channels, lowest accuracy).

[0068] Embodiments disclosed herein provide an audio processing apparatus and method implementing Mixed-Order Stereophony over Headphones (MOS-Hp) for rendering 3D binaural audio for headphone reproduction. MOS-Hp implemented by an audio processing apparatus and method according to an embodiment may be used to render 3D audio for a wide range of audio inputs (objects, upmixing of legacy stereo / surround sound, real microphone array signals) and with dynamic head-tracking. It compresses any number of audio objects into a fixed number of channels.

[0069] As will be described in more detail below, MOS-Hp uses a similar rendering chain to HOA and HOS, but provides a kind of middle ground between these two audio processing scheme. In other words, MOS-Hp can utilise the benefits of HOS (fewer channels needed for rendering compared to HOA) whilst overcoming the drawbacks of HOS (lower accuracy compared to HOA). As will be described in more detail, an audio processing apparatus according to an embodiment implementing MOS-Hp achieves this by utilising a similar approach to HOS however allowing for a selectable or user-defined number of audio channels to be used in the rendering stage (channel count set between the lower HOS and upper HOA limits of (N+l) to (A + I)2). This means that an audio processing apparatus according to an embodiment implementing MOS-Hp makes use of the advantage of the HOS sparse representation of the soundfield and HRTF, meaning less channels compared to HOA are needed. Moreover, an audio processing apparatus according to an embodiment implementing MOS-Hp overcomes the drawback of HOS that the number of reproduced audio channels is fixed to (N+l) which whilst minimises computational load, can lead to unacceptable error in the accuracy of the soundfield. Furthermore, an audio processing apparatus according to an embodiment implementing MOS-Hp overcomes the disadvantage of HOA that a large number of audio channels, ((A + I)2) are needed for high accuracy.

[0070] Figure 5 is a schematic diagram illustrating an audio processing apparatus 100 according to an embodiment implementing MOS-Hp for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones 120. As will be described in more detail in the following, MOS-Hp uses a mathematical basis called spherical harmonics to perform a decomposition or representation of the soundfield (this is the same framework as used by HOS and HOA ensuring compatibility with conventional audio content). The basis decomposition requires infinite number of terms for complete accuracy, as this is not physically possible the decomposition is truncated to an order NMAX, which limits the number of audio channels to a finite number but means the soundfield is only accurately rendered below a high frequency limit.

[0071] As will be described in more detail in the following, the audio processing apparatus 100 according to an embodiment implementing MOS-Hp is configured in a processing stage to encode one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis such that the spherical harmonic representation of the one or more input audio signals comprises a subset of the (NMAX+1)2spherical harmonic channels and the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener. Subsequently, in a further processing stage the audio processing apparatus 100 according to an embodiment implementing MOS-Hp is configured to generate a left ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a left ear HRTF and / or HRIR and to generate a right ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR.

[0072] In an embodiment, the audio processing apparatus 100 may be further configured to select the subset of the (NMAX+1)2spherical harmonic channels, as will be described in more detail below. In an embodiment, the audio processing apparatus 100 according to an embodiment may further perform a rotation of the soundfield to compensate for listener head rotations (this operation is also used in HOA Tenderers). In an embodiment, the audio processing apparatus 100 according to an embodiment may further perform a rotation of the soundfield so that the y axis of the spherical harmonics reference system 600 is rotated to the z axis so that the interaural axis of the listener, i.e. the line running through the listener’s ears 601a, b, coincides with the z axis of the spherical harmonics reference system 600, as illustrated in figure 6. As will be appreciated, the further rotation illustrated in figure 6 corresponds to the “HOS rotation” implemented in HOS Tenderers. As already described above, in a further processing stage the audio processing apparatus 100 according to an embodiment is configured to select the spherical harmonic channels. In an embodiment, this selection may be implemented by first encoding the input signal using all (NMAX+1)2spherical harmonic channels and removing those channels not required in the final MOS format (reduction of audio channels to a user specified value). As already described above, in a further embodiment, the audio processing apparatus 100 may perform the encoding using only the selected subset of the (NMAX+1)2spherical harmonic channels. As will be appreciated, for reproduction to order N, less audio channels and operations are required with MOS-Hp compared to HOA, but more channels and operations are required with MOS-Hp than HOS. For reproduction to order N, any number of user specified channels between N + 1 and (A + I)2may be used. This means the optimisation is adaptive to use case - the user may choose the number of audio channels and decide the trade-off between computational efficiency and accuracy of the rendered soundfield. MOS-Hp leads to a higher accuracy than HOS, but can never exceed then accuracy of HOA. For instance, for N=4, HOA would use 25 channels, while HOS would use 5 channels. The audio processing apparatus 100 according to an embodiment would use between 5-25 channels depending on the design scenario and / or user selection. If instead the number of audio channels if fixed, MOS-Hp as implemented by the audio processing apparatus 100 according to ab embodiment may reproduce to a higher order N than HO A which can lead to the soundfield being recreated accurately to a higher frequency than HOA.

[0073] In an embodiment, the audio processing apparatus 100 may be configured to perform the selection, i.e. decimation step based on a decimation matrix D. This may be related to a HRTF optimisation process which includes further processing to improve the performance of the HRTF above the high frequency accuracy threshold. This operation may be performed offline before the audio processing apparatus 100 begins rendering input audio and is only performed once on the HRTF. The HOS rotation leads to a sparse representation of the head-related transfer function (HRTF) when using spherical harmonics. The MOS representation uses this rotation to create a more complete and accurate representation of the HRTF compared to HOS. Because of this, less audio channels and less filter convolutions are required when rendering the audio in real-time compared to HOA, but with a comparable level of accuracy.

[0074] This is achieved by the following operations. A HRTF measured at (A + I)2positions distributed across a sphere is provided as input. The ears of the HRTF are assumed to lie along the cartesian Y axis. The HRTF may be pre-processed using equalisation to later improve performance, in particular to minimise error in the reproduced soundfield above the high frequency threshold. This can include removing interaural time differences above a certain frequency (time alignment) or utilising a zero-phase filter transformation, as will be described in more detail below. The HRTF may be transformed into a spherical harmonic representation using the pseudoinverse, which gives the (A + I)2HRTF spherical harmonic coefficients (each coefficient becomes an audio channel during the rendering). The HRTF spherical harmonic coefficients are rotated so that the ears of the HRTF lie along the cartesian Z axis (referred to as the HOS rotation above). This operation reorders the energy of the spherical harmonics into those close to spherical harmonics with m=0. This is shown in figures 7a and 7b, where the m=0 channels are the center of the pyramid. Figures 7a and 7b show the HRTF spherical harmonic coefficients before (figure 7a) and after (figure 7b) the rotation for N=30. Each pixel represents the energy in the coefficient / channel. Light gray is high energy, while dark grey is low energy. High energy is assumed to mean that spherical harmonic channel has significant information and is kept for rendering. Before the rotation the energy is spread significantly across all coefficients. After the rotation the energy is focused into a smaller number of channels around the spherical harmonics with degree m=0. Then, the decimation matrix is applied, where a number of the spherical harmonic coefficients / channels are removed, i.e. the subset of channels is selected. This decision may be made based on energetic analysis (those with low energy are removed), perceptual motivations (e.g. from listening test observations) or other motivations. The coefficient count may be reduced from (IV + I)2to any user specified number between N + 1 and (IV + I)2depending on the use case. This is where the approach differs from HOS, which has the restriction that only the m=0 channels are kept. However, this extreme decimation leads to the removal of a significant amount of energy and therefore accuracy in the final 3D audio binaural signals. Keeping all (IV + I)2channels leads to identical final 3D audio binaural signals as with a HOA Tenderer. Thus, the decimation step involves deciding which coefficients may be discarded, indicated by the black pixels shown in figure 7. The assumption is the discarded channels will not reproduce a significant amount of information in the final rendering stage, therefore may be discarded to increase computational efficiency (at the cost of a slight reduction in accuracy).

[0075] Further equalization may then be performed on the spherical harmonic coefficients to mitigate any further artifacts. The HRTF is now in MOS format and ready to be used by the audio processing apparatus 100 for rendering input audio. As will be appreciated, the HRTF optimization process is used to define which spherical harmonic channels are kept. The set of MOS rotation and decimation matrices are then be applied to the HRTF (once, before runtime), and to the input audio / soundfield (continuously, during runtime).

[0076] Referring to figure 5 again, the audio processing apparatus 100 according to an embodiment implementing MOS-Hp rendering accepts object audio input or HOA B-format signals. For audio objects each mono audio object may be converted to the multichannel MOS format by the MOS Object Encoder 103. The signal processing parameters required for this operation are based on the Object Metadata and Head-tracking Data, which are processed by the Encoder Metadata Controller 101. This operation is repeated for each audio object and all the resulting MOS format signals are summed. Legacy channel-based content (e.g. 5.1) may be inputted by creating a virtual loudspeaker array using the object input. Mathematically the operation performed by the MOS Object Encoder 103 to encode all the mono audio objects in the frequency domain (operation repeated for each individual frequency bin), is as follows: with btbeing the z-th object signal, Ytthe vector of all the (IV + I)2spherical harmonics up to order N sampled at the z-th source position and bH0Athe resulting HOA B-format signals summed over all audio object inputs. Next, the signals are transformed into the MOS format, b: b = D RzRbH0Awith R the rotation of the soundfield to compensate for listener head rotations, Rzthe rotation that rotates the y axis to the z axis, D the decimation matrix that removes the spherical harmonic channels not required as specified by the HRTF optimization. This leaves a multichannel audio signal b of size between (A + 1) to (A + I)2which is the MOS format.

[0077] A similar set of operations is performed by the HOS Rotation and MOS Decimation block 104 to transform HOA B-Format input signals into MOS format signals, to be combined with the MOS signals generated as output of the MOS Object Encoder 103.

[0078] Note the same set of operations is used for the conventional HOS Tenderer, however for the MOS approach the decimation matrix D differs due to the HRTF optimization, which may be described mathematically as follows: hlir= W2D RJ^pWiHRTFi r

[0079] Here is the vector of spherical harmonic coefficients of the optimized HRTF. To achieve this, the input measured HRTFs (minimum number of measured positions (A + I)2), contained in the HRIR Dataset 105 may be formatted into a vector, HRTF. w is the first equalization operation, performed by the HRIR Zerophase Optimisation 106, for example using the zero-phase approximation of the HRTF above a certain frequency threshold, described in more detail below. YHRTFis a matrix of spherical harmonics sampled at the measured HRTF positions. The pseudoinverse of this matrix multiplied with the sampled HRTF transforms it into its spherical harmonic representation. Rzand D are identical operations as to the soundfield, introduced above, to rotate the y axis to the z axis (the HOS rotation) and perform decimation (MOS decimation) to remove unwanted channels, respectively. These operations are performed by the HRIR HOS Rotation and MOS Decimation block 107. w2is a final equalization stage (not shown directly in the figure, as it may be integrated in block 107). The HRTF is now in its optimized MOS format, with the indexes l,r indicating the operations must be performed separately for the left and right ears in turn. This whole operation chain may be performed once before runtime.

[0080] The binauralisation, performed by the Binauralisation block 108, combines the HRTF and the soundfield both in MOS format: resulting in binaural signals, for the left and the right ear of the listener to be reproduced over the headphones 120. Again, all operations above are performed in the frequency domain and independently for each frequency. An analogous set of operations in the time domain can be derived straightforwardly from these operations, where multiplications become convolutions.

[0081] As already described above, in an embodiment the audio processing apparatus 100 comprises a processing block 106 for HRIR zero-phase optimization. To this end, for frequencies above a frequency threshold value or limit, the audio processing apparatus 100 according to an embodiment is configured to generate the left ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the left ear HRTF and / or HRIR and the right ear binaural audio rendering signal based on the subset of the plurality of spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the right ear HRTF and / or HRIR. In an embodiment, the audio processing apparatus 100 may filter the left ear HRTF and / or HRIR and the right ear HRTF and / or HRIR with an all-pass filter for generating the zero-phase filter representations.

[0082] As will be appreciated, the zero-phase optimization implemented by the audio processing apparatus 100 according to an embodiment addresses the fundamental issue of using the spherical harmonic decomposition of a HRTF and soundfield that above the high frequency threshold value or limit, the representation of the HRTF / soundfield is no longer accurate. This limit is set by the truncation order N, where a higher truncation order leads to more audio channels / computational load but more accuracy. The zero-phase optimization 106 implemented by the data processing apparatus 100 according to an embodiment performs a pre-processing of the HRTF before it is represented using the spherical harmonic basis and, thus, allows making the final binaural signals more perceptually accurate to reproduce the desired 3D audio scene.

[0083] As already described above, the pre-processing implemented by the block 106 may use a special filter representation (herein referred to as the zero-phase filter). Zero-phase filters are generally known, but have not been applied to HRTFs or binaural audio reproduction yet. As used herein, a zero-phase filter has a phase that has a fixed value of 0 or TI. Applying the zerophase transform to the HRTF filter leads to a zero-phase filter HRTF which has identical magnitude to the original HRTF, but a phase fixed at 0 or TI for each frequency. This has the effect of at high frequencies allowing the audio processing apparatus 100 to better render more accurately the magnitude of the soundfield using less audio channels, as at high frequencies phase information is less important for 3D audio. Therefore, the optimization process implemented by the processing block 106 of the audio processing apparatus 100 creates a perceptually more accurate experience for the end user and attempts to overcome the effect of the high frequency accuracy limit.

[0084] More specifically, the zero-phase filter approach may be implemented by the audio processing apparatus 100 according to an embodiment in the following way. The HRTF is measured for a minimum of (A + I)2positions, which are formatted into two vectors formatted into a vector one for the left and one for the right ear, HRTFir. The procedure is performed independently for the left and right ears. To estimate the spherical harmonic coefficients of the HRTF:

[0085] ^l,r =1HRTF HRTFl rwith YH^RTFthe pseudoinverse of a matrix of spherical harmonics is sampled at the HRTF sampling positions. This form of hir, the vector of spherical harmonic coefficients of the HRTF, may lead to inaccuracies at high frequencies. The zero-phase approximation of the sampled HRTF vector is also calculated forming the (IV + I)2length vector fj^'pFierrophase. The zero-phase HRTF spherical harmonic coefficients are then calculated as follows:

[0086] The final set of HRTF spherical harmonic coefficients used in the audio processing apparatus 100 according to an embodiment use the standard approach at low frequencies, and the zerophase approach for frequencies higher than the high frequency threshold value, i.e. limit. Thus

[0087] In an embodiment, the change from hL rto hz^rophasemay asudden cut off, but may be slowly faded between to minimise artefacts. The set of h°ptimisedforms the optimised HRTF filters ready for use in the audio processing apparatus 100 for improved 3D audio rendering.

[0088] Figure 9 is a flow diagram illustrating an audio processing method 900 for generating binaural audio rendering signals for audio rendering one or more audio input signals over the headphones 120 worn by a listener. The audio processing method 900 comprises a step 901 of encoding the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signal comprises a subset of the (NMAX+1)2spherical harmonic channels and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener. Moreover, the audio processing method 900 comprises a step 903 of generating a left ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a left ear HRTF and / or HRIR and generating a right ear binaural audio rendering signal based on the subset of the plurality of, i.e. the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR. The audio processing method 900 can be performed by the audio processing apparatus 100 according to an embodiment. Thus, further features of the audio processing method 900 result directly from the functionality of the audio processing apparatus 100 as well as its different embodiments described above and below.

[0089] The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the present disclosure (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step).

[0090] In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described embodiment of an apparatus is merely exemplary. For example, the unit division is merely logical function division and may be another division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0091] The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

[0092] In addition, functional units in the embodiments of the invention may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Claims

CLAIMS1. An audio processing apparatus (100) for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones (120) worn by a listener, wherein the audio processing apparatus (100) is configured to: encode the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signals comprises a subset of (NMAX+1)2spherical harmonic channels and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener; and generate a left ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a left ear head-related transfer function, HRTF, and / or head-related impulse response, HRIR, and a right ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR.

2. The audio processing apparatus (100) of claim 1, wherein the audio processing apparatus (100) is further configured to select the subset of the (NMAX+1)2spherical harmonic channels.

3. The audio processing apparatus (100) of claim 2, wherein the audio processing apparatus (100) comprises a user interface, wherein the user interface is configured to receive a user input and wherein the audio processing apparatus (100) is configured to select the subset of the (NMAX+1)2spherical harmonic channels based on the user input.

4. The audio processing apparatus (100) of claim 2 or 3, wherein the audio processing apparatus (100) is configured to obtain or estimate an energy associated with each of a plurality of spherical harmonic channels of a spherical harmonic representation of the left ear HRTF and / or HRIR and an energy associated with each of a plurality of spherical harmonic channels of a spherical harmonic representation of the right ear HRTF and / or HRIR and to select the subset of the (NMAX+1)2spherical harmonic channels of the spherical harmonicrepresentation of the one or more input audio signals based on the energy of each spherical harmonic channel of the spherical harmonic representation of the left ear HRTF and / or HRIR and the energy of each spherical harmonic channel of the spherical harmonic representation of the right ear HRTF and / or HRIR.

5. The audio processing apparatus (100) of any one of claims 2 to 4, wherein the audio processing apparatus (100) is configured to select the subset of the (NMAX+1)2spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals based on the degree of the spherical harmonic coefficients of the spherical harmonic representation of the left ear HRTF and / or HRIR and / or the degree of the spherical harmonic coefficients of the spherical harmonic representation of the right ear HRTF and / or HRIR, wherein the reference axis of the spherical harmonic representation coincides with the listener’s interaural axis.

6. The audio processing apparatus (100) of any one of the preceding claims, wherein for encoding the one or more input audio signals into the spherical harmonic representation based on the plurality of spherical harmonics from order 0 to order NMAX and defined relative to the reference axis the audio processing apparatus (100) is configured to: encode the one or more input audio signals into a preliminary spherical harmonic representation based on the plurality of spherical harmonics from order 0 to order NMAX and defined relative to a vertical axis, wherein the preliminary spherical harmonic representation of the one or more input audio signals comprises the subset of the (NMAX+1)2spherical harmonic channels and wherein the vertical axis of the preliminary spherical harmonic representation does not coincide with the interaural axis of the listener; and rotate the preliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals based on an orientation of interaural axis of the listener relative to the vertical axis.

7. The audio processing apparatus (100) of claim 6, wherein the reference axis of the spherical harmonic representation defines the z-axis of a reference system of the spherical harmonic representation and wherein the interaural axis of the listener lies in the xy-plane of the reference system of the spherical harmonic representation, wherein for rotating thepreliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals based on the orientation of interaural axis of the listener relative to the reference axis the audio processing apparatus (100) is configured to: rotate the preliminary spherical harmonic representation of the one or more input audio signals into a further spherical harmonic representation of the one or more input audio signals so that the interaural axis of the listener coincides with the y-axis of the reference system of the spherical harmonic representation; and rotate the further preliminary spherical harmonic representation of the one or more input audio signals into the spherical harmonic representation of the one or more input audio signals so that the interaural axis of the listener coincides with the z-axis of the reference system of the spherical harmonic representation.

8. The audio processing apparatus (100) of any one of the preceding claims, wherein the number NCH of spherical harmonic channels of the subset of the (NMAX+1)2of spherical harmonic channels of the spherical harmonic representation of the one or more input audio signals is larger than NMAX+1 and smaller than (NMAX+1)2.

9. The audio processing apparatus (100) of any one of the preceding claims, wherein the audio processing apparatus (100), for frequencies below a frequency threshold value, is configured to generate the left ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the left ear HRTF and / or HRIR and the right ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the right ear HRTF and / or HRIR and wherein the audio processing apparatus (100), for frequencies above the frequency threshold value, is configured to generate the left ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the left ear HRTF and / or HRIR and the right ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a zero-phase filter representation of the right ear HRTF and / or HRIR.

10. The audio processing apparatus (100) of claim 9, wherein for generating the left ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the zero-phase filter representation of the left ear HRTF and / or HRIR and the right ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and the zero-phase filter representation of the right ear HRTF and / or HRIR the audio processing apparatus (100), for frequencies above the frequency threshold value, is configured to filter the left ear HRTF and / or HRIR and the right ear HRTF and / or HRIR with an all-pass filter.

11. Headphones (120) for audio rendering a left ear and right ear binaural audio rendering signal, wherein the headphones comprise an audio processing apparatus (100) according to any one of the preceding claims.

12. An audio processing method (900) for generating binaural audio rendering signals for audio rendering one or more audio input signals over headphones (120) worn by a listener, wherein the audio processing method (900) comprises: encoding (901) the one or more input audio signals into a spherical harmonic representation based on a plurality of spherical harmonics from order 0 to order NMAX and defined relative to a reference axis, wherein the spherical harmonic representation of the one or more input audio signal comprises a subset of (NMAX+1)2spherical harmonic channels and wherein the reference axis of the spherical harmonic representation coincides with an interaural axis of the listener; and generating (903) a left ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a left ear head-related transfer function, HRTF, and / or head-related impulse response, HRIR, and a right ear binaural audio rendering signal based on the subset of the (NMAX+1)2spherical harmonic channels of the one or more input audio signals and a right ear HRTF and / or HRIR.

13. A computer program product comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method (900) of claim 12 when the program code is executed by the computer or the processor.

Images