Method for generating an audio scene in a binaural spatialization system

Abstract

A method for generating an audio scene in a binaural spatialization system, comprising a step of providing a set of primary auditory transfer functions, relating to the left and right ears of individuals, recorded from a set of N individuals; a preliminary preparation step, with conversion to logarithmic values ​​and symmetrization of the spectral amplitude; a step of smoothing the spectral amplitude by binaural relative deviation, normalization for a reference direction (θ0,Φ0) with average across subjects, with flattening of the amplitudes on the ipsilateral side on a frontal cap (CF); a step of evaluating the mean interaural delay; and a step of generating a set of reprocessed transfer functions to generate an audio scene in real time using a binaural rendering engine, with one or more point audio sources at prescribed position(s). Figure from the abstract: Fig. 6

Description

Title of the invention: Method for generating an audio scene in a binaural spatialization system

[0001] The invention relates to a method for generating an audio scene in a binaural spatialization system. The invention also relates to a system and headphones in which the proposed method can be implemented.

[0002] In general, it is becoming increasingly common to need to generate a three-dimensional audio scene by synthesis, whether to reproduce stereophonic music listening on headphones or via earplugs or in the case of immersion scenarios in a virtual reality with a virtual reality headset.

[0003] Binaural hearing is when both ears of an individual receive specific sound signals, corresponding to a perception of sounds in space allowing the auditory system and the brain of the individual to determine the direction from which a sound originates.

[0004] Thus, thanks to binaural hearing, spatialized sound reproduction allows a listener to perceive sound sources coming from any direction or position in space, and to have the impression of being immersed in a three-dimensional sound scene.

[0005] The signals received at the right and left ears of the listener are differentiated, in particular but not exclusively, by their intensity and by the delay of arrival and phase.

[0006] The term "multichannel," in the context of spatialized sound reproduction, refers to producing a representation of an acoustic scene in the form of P signals (called spatial components). These signals contain all the sounds that make up the sound scene, but with weightings that depend on their direction (or "incidence") and are described by P associated spatial rendering functions. It should be noted, however, that the proposed method also works for a scene with a single audio source.

[0007] The specific spatialized sound reproduction techniques to which the present invention relates are based on the existence of acoustic transfer functions of the head between spatial positions and the ear canal. These transfer functions, called "HRTFs" (for "Head Related Transfer Functions"), concern the frequency form of the transfer functions. Their time-domain form will be referred to hereafter as "HRIRs" (for "Head Related Impulse Response").

[0008] More specifically, we are particularly interested in a set of transfer functions auditory including for a given individual and for a plurality of measured positions M: a frequency transfer function for the left ear (HRTFg (M)(f)), a frequency transfer function for the right ear (HRTFd(M)(f)), an impulse response function for the left ear (HRIRg(M)(t)) and an impulse response function for the right ear (HRIRd(M)(t)).

[0009] Public databases exist in which these measurements of auditory transfer functions have been listed for a plurality of individuals. In the remainder of this document, a particular individual will be assigned the index 'i' and the number of individuals will by convention be equal to N.

[0010] The ipsilateral ear is the ear that receives the sound first, and the contralateral ear is the ear that receives the sound second for the same sound source. In the remainder of this document, the left hemisphere is the hemisphere located to the left of the sagittal plane referenced by the listener's head in an upright position, and the right hemisphere is the hemisphere located to the right of this sagittal plane.

[0011] The left ear is the ipsilateral ear for sounds coming from the left hemisphere and the right ear is the ipsilateral ear for sounds coming from the right hemisphere.

[0012] For the generation of a sound scene by synthesis in a binaural rendering engine, a convolution operation is carried out between the sound sources of the scene and the temporal impulse responses, respectively right and left, or if working in the frequency domain, a multiplication operation of the spectra followed by an inverse Fourier transform.

[0013] It turns out that using a set of auditory transfer functions measured on a particular individual is not satisfactory if used for a general binaural rendering engine used for a general audience.

[0014] Building an average of measurements over N individuals and producing a set of average auditory transfer functions does not give complete satisfaction either.

[0015] Some have tried to work on the phase of the transfer functions as for example in document US 10609504 but the subjective results still leave something to be desired.

[0016] The inventors sought to improve the situation, notably by reworking the transfer function sets from public database(s), with a particular focus on binaural rendering in headphones. Special attention is paid to reproducing the timbre of the audio sources and eliminating possible comb filter effects.

[0017] The proposed method and system can also be used in a so-called "transaural" context with listening on loudspeakers.

[0018] The inventors also maintained the objective of being able to offer a possibility of individualization with respect to the listener who will be immersed in the three-dimensional audio scene.

[0019] To this end, a method for generating an audio scene in a binaural spatialization system is therefore proposed herein, the method being characterized in that it comprises: S0 - a step of making available a set of primary (basic) auditory transfer functions, relating to the left and right ears of individuals, recorded on a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) denoted $ ) Ct including amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) denoted H6j (fi ) ct including amplitude and phase, a left-ear impulse response time function (HRIRg) (i,Mj)(t)) denoted and an impulse time response function for the right ear (HRIRd (i,Mj)(t)) denoted For each position Mj having polar coordinates 07 and ¢ (azimuth and elevation), the process includes: SA- a preliminary preparation step noted SA, in which a conversion to logarithmic values ​​is carried out, with, where applicable (if this has not already been done in the public database), a symmetrization of the spectral amplitude for all directions (d (fi ) of a subject i, using an averaging function FA, according to the formulation: -FA dB^-O^))

[0020] R'(0^) = FA [dB (h^, dB (H^- 0^ (fi^)

[0021] the function dB being the decimal logarithm function, the function FA being able to be in a simplified case the half sum, SB, a step denoted SB, involves smoothing the spectral amplitude by binaural relative deviation, normalization for a reference direction with average across subjects. according to the wording:

[0022] =

[0025] dB(H r(B, ¢)) = B, 0. ¢)) + dB(Hl(0, (¢))

[0024] SD- a step marked SD, for evaluating the mean intraural delay; = argmax ( |Ag(0, ¢) *vv| )

[0025] — ±argmax( |hld*w| )

[0026] t{ 0, U

[0027] fs being the sampling frequency of the HRIR impulse responses, and argmax(un) being the index of the maximum of a sequence u”, of digitized values ​​of the impulse response, the notation * being the convolution product, w being a convolution filtering function, which can be reduced to a Dirac pulse (then we simply select the largest signal peak),

[0028] SF- a step denoted SF, of generation of a set of reprocessed transfer functions, denoted HHi and HHr, HH^O, ¢) = V20

[0029] HHr{ 0^) =

[0030] SR- a step denoted SR of using the reprocessed functions HH[(0, and HHr( 0, ( / >) to generate in real time an audio scene using a binaural rendering engine, with one or more point audio source(s) at prescribed position(s).

[0031] Thanks to the provisions described above, the development and use of the reprocessed transfer functions yields very satisfactory results. The timbre of the audio source or sources is reproduced very satisfactorily for a group of users in a tested panel.

[0032] It is further noted that the normalization of the amplitudes of spectral filters, with respect to a reference direction j, proves to be particularly beneficial, especially from the point of view of general balancing.

[0033] Advantageously, it is noted that the comb filter effects are eliminated.

[0034] The average effect with respect to the set of individuals measured makes it possible to work on an average individual and to maximize the suitability of the reworked transfer functions with respect to the general population able to use devices implementing the proposed process.

[0035] It should be noted that the number of audio sources participating in the generation of synthesized audio signals is arbitrary. A single audio source may correspond to a videoconference with a person delivering a speech in a location without any other sound source in the audio scene. In other configurations, a multi-layered sound environment is provided, with multiple sound sources located in front of and behind the listeners, as in the case of immersion in a first-person shooter (FPS) video game or in the case of simulators of all types.

[0036] It should also be noted that, according to the proposed method, calculations are performed in logarithmic form and then, conversely, in linear form before the filters, with their amplitudes and phases, are injected into the binaural rendering engine. This allows for subtractions instead of divisions, thus reducing the overall computational load.

[0037] As will be specified later, certain steps are carried out in an offline mode, that is to say, prior to actual use in a rendering engine Binaural processing occurs in an audio playback device, while conversely, certain steps are performed in real time, namely during the generation of sound waves within the audio playback device. As we will see, there are several possible solutions for the functional separation between offline and real-time processing.

[0038] According to one aspect, the process may further comprise a step denoted SB2 amplitude flattening of filters on a frontal cap (CFg) for the ear ipsilateral, according to the formulation: for any direction 07, <h, comprise dans the solid angle defined by the respective bounds [0,00] and [- <b0 +<b0] côté gauche. La The formula is given for a source located on the left. If the source is located on the right then dB(Hr( dB(Hr( da 0()) ) for any direction 07, <h, comprise in the solid angle CFd defined by the respective bounds [0,00>] and [- <b0 + 0] right side.

[0039] Advantageously, the amplitudes applied for flattening correspond to the amplitudes of the filter in the reference direction OqOq on the ipsilateral side. According to one option, the filters for the contralateral ear remain unchanged, particularly with regard to the amplitudes and minimum phases.

[0040] Accordingly, a flat, uncorrected filter is applied to the amplitude of the filter of the ipsilateral ear, within a frontal cap. The trials and tests conducted by the inventors revealed that this arrangement was favorable from the point of view of perceived quality for the users of the test panel.

[0041] It is noted that, at the boundaries of the cap, there is no jump or discontinuity, because the amplitudes have previously been normalized with respect to the reference amplitude in the direction [OoOoL The filter is continuous and differentiable over the entire range of angles.

[0042] According to another formulation of this step, the step denoted SB2 of flattening the amplitude of the filters on a front cap (CF), can be carried out according to the formulation: 100431 dB(H,(". 0) ) = ¢) - A l»|. *)) dB(Hr(e, (R\e, ¢)¢) )

[0044] It is specified here that this step of the process flattens the HRTF on the source side, in this case the left-hand side. Therefore, the same logic is to be applied symmetrically for a source located on the right.

[0045] According to one aspect, the binaural rendering engine further includes a head tracking function. In practice, a 3-axis gyroscope provides the yaw, roll, and pitch movements to headphones on which the binaural rendering engine is implemented. It should be noted that the gyroscope can be integrated into the headphones. Alternatively, the yaw movement can be determined using one or more cameras, whether or not they are integrated into the headphones. Thus, the rendering engine can compensate in real time for the listener's head movements by retrieving the corresponding inverse positions from the database: e.g., if a listener turns their head 15° to the left, a sound initially positioned at 0° in front of them will shift 15° to their right.

[0046] According to one aspect, the process may further include, before the SF step or during the SR step, a step denoted SE of interpolation of spectral amplitudes, phases and internal delays.

[0047] Wherefore, starting from a basis of known transfer function sets with a sufficiently large step, for example every 15° or every 10°, it is possible to provide, through interpolation, a basis of transfer function sets with better granularity, for example every 3° or even every 2°.

[0048] It should be noted that this interpolation step can be carried out in the offline stages or can be carried out in real time with the binaural rendering engine.

[0049] According to one aspect, the spectral amplitude symmetrization (SA) step may further include smoothing by local average of k neighboring directions j, according to the formulation: L\9, #)-^(^(^(^ ef ) R* (ft #) = 4 EJ (dB ((Bj, ¢.)) + dB (j))

[0050] This smooths the filters not only on individuals, but also in space.

[0051] According to one aspect, the method may further include a step of individualizing the interaural delay (SG) as a function of a head radius b of a target listener, the interaural delay being determined by with b the new head radius and a the original radius, i.e. the reference head radius.

[0052] According to one aspect, the method may further comprise an angular correction step, denoted SH, in fact an azimuthal angular correction, with a parameter y that can be adapted to each user, according to the formulation:

[0053] / (0,0) = (0 + y*sin(20),0)

[0054] The parameter is preferably configurable, and can even be individualized according to the listener benefiting from the binaural audio rendering.

[0055] In other words, instead of using the azimuth angle 3 directly, we use instead a transform of the azimuth angle in the form 0 + y*sin(20) • H. It turns out that the qualitative result is further improved by this angular deformation.

[0056] According to a related aspect, the method may include an angular correction step denoted SH with an angular correction in azimuth and elevation where the angles 0,q> in the HRTF formulas are replaced by the angles (0',q / ) such that (0',q / ) = f (0,q>).

[0057] According to an example, we can choose (0', 0') = f(0, 0) = (0 + y*sin(20), 0 + y'*sin(0)).

[0058] The parameters y and y' can be adapted to each user.

[0059] In other words, instead of using the azimuth angle 8 directly, we use instead a transform of the azimuth angle in the form 0 + y*sin(20) and instead of using the elevation angle q> directly, we use instead a transform of the elevation angle in the form 0 + y'*sin( 0) •

[0060] According to one aspect, the process may further comprise a step denoted SC of symmetrization and phase averaging, according to the formulation: <W)> = + ) 100611)

[0062] The notation < . > denotes the phase of the complex transfer function.

[0063] This further improves the averaging function of the individuals who participated to the campaigns to measure transfer functions.

[0064] The invention also relates to headphones configured to perform steps SE to SR of a process as described above. Depending on the configuration, the headphones can perform steps SG to SR or SH to SR.

[0065] The invention also relates to a processing system configured to implement steps S0 to SG of a process as described above.

[0066] According to another embodiment of the present invention, starting from certain HRTF filter bases, possibly already symmetrized, the flattening defined in step SB2 is simply applied. In this case, it may be necessary to connect the filters by continuity to the edge of the CFg (respectively CFd) cap to guarantee the absence of a jump. This can be done by polynomial interpolation.

[0067] The invention will be further detailed by describing non-limiting embodiments, and based on the accompanying figures illustrating variants of the invention, in which: - [Fig.l] is a schematic diagram illustrating the spherical coordinates of points and directions in space around an individual, for example a listener in the context of listening or in the context of measuring auditory transfer functions HRTF; - [Fig.2] illustrates an example of the amplitude of a frequency transfer function, with the phase, for the contralateral ear with normalization with respect to the 15° direction; - [Fig.3] illustrates an example of the amplitude of a frequency transfer function, with additional phase for the ipsilateral ear with normalization relative to the 15° direction; - [Fig.4] illustrates an example of the amplitude of a frequency transfer function, with additional phase for the contralateral ear with normalization relative to the 30° direction; - [Fig. 5] illustrates an example of the amplitude of a frequency transfer function, with the phase, with the phase for the ipsilateral ear with normalization relative to the 30° direction, and flattening on the frontal cap; - [Fig.6] is analogous to [Fig.1] and illustrates in particular areas of the ice cap frontal where the filtering spectra are flattened; - [Fig.7] schematically illustrates an audio scene reproduction using a binaural rendering engine, for a single audio source ASi; - [Fig.8] schematically illustrates an example of the sequence of steps of the proposed process according to a first variant of hardware implementation; - [Fig.9] schematically illustrates an example of a succession of steps in the proposed method according to a second variant of hardware implementation; - [Fig. 10] schematically illustrates an example of the sequence of steps in the proposed process according to a third variant of hardware implementation.

[0068] In the different figures, the same reference numerals designate identical or similar elements. For clarity, some elements are not necessarily shown to scale.

[0069] Generalities and system

[0070] Generally, spatialized binaural sound reproduction allows a listener U to be immersed in an audio scene and to perceive sound sources coming from different directions and positions in space, while the listener U only has two speakers HPg and HPd.

[0071] In the example of [Fig.7], the playback device is an audio headset 2 with the left speaker HPg arranged opposite the left ear OG of the listener and the right speaker HPd arranged opposite the right ear OD of the listener U.

[0072] In other configurations, the reproduction organ can be a pair of intra-aural earphones such as, for example, "earplugs".

[0073] In yet another configuration, in a so-called open-air listening context sometimes referred to as 'transaural™', two reproduction devices, such as conventional loudspeakers, are arranged with crosstalk cancellation. The proposed invention can also be applied in this transaural configuration, with two or more loudspeakers.

[0074] For the different configurations mentioned above, the two speakers can be controlled by a single control unit directly in wired mode, or the speakers can be controlled by a single control unit but via a wireless Bluetooth connection.

[0075] In one scenario, the audio scene to be reproduced may consist of a single sound source, denoted ASi in [Fig. 7]. In another scenario, the audio scene to be reproduced consists of a plurality of sound sources ASj. For example, the audio scene to be reproduced may include NbS sound sources.

[0076] Each sound source to be reproduced, such as the first source ASi, is located at a position Ml characterized by its polar coordinates 01 and ¢1 and its distance rl, as illustrated in [Fig. 1]. Polar coordinates will be used throughout this document. The first coordinate 0 represents the azimuth with respect to a "straight ahead" direction, with positive angles being denoted to the right or left depending on the side of the source concerned. The second coordinate (also noted <j>) corresponds to the elevation, also called "site", relative to a horizontal direction, with positive angles being pointed upwards.

[0077] In [Fig. 1], the plane labeled PH is the horizontal reference plane for the listener with their head upright. The plane labeled PMS is the median sagittal plane, separating the left hemisphere from the right hemisphere. The plane labeled PVI is the vertical interaural plane (passing through 0° = 90°). When the system is equipped with the head-tracking function, the deviation of the actual position from the upright head position is compensated by the same angle values, but with opposite signs. For the listener, the sound source always appears to be in its given position in space, regardless of any small head movements the listener may make.

[0078] The reproduction of an audio scene is based on the use of acoustic transfer functions of the head between the positions in space (0,0) and the ear canal, called auditory transfer functions. These transfer functions, known as "HRTFs" (for "Head Related Transfer Functions"), concern the frequency form of the transfer functions. Their time form, that is to say the impulse response, will be referred to hereafter as "HRIR" (for "Head Related Impulse Response").

[0079] In the context of binaural playback, transfer functions are then used following: a frequency transfer function for the left ear (HRTFg(M)(f)), a frequency transfer function for the right ear (HRTFd(M)(f)), an impulse response function for the left ear (HRIRg(M)(t)) and an impulse response function for the right ear (HRIRd(M)(t)), where M is the position of the source in polar coordinates, f is the frequency and t is the time.

[0080] Advantageously, according to the present invention, instead of using standard transfer functions, after modifying and improving these transfer functions, modified transfer functions, denoted in this document HHi and HHr, are used. Their frequency form will be denoted HH] and HHr, and their time form hh] and hhr. The method for obtaining these modified transfer functions will be explained in detail later in this document.

[0081] Returning to [Fig. 7], the audio scene is rendered by a binaural rendering engine. The binaural rendering engine has at its disposal the transfer functions processed in their frequency or time form. Thanks to these functions, for each sound source ASi of the audio scene to be reproduced, characterized by the time-wave pattern ASi(t) of the sound source as well as its polar coordinates (0i, <e>i) and its distance from the listener ri, the binaural rendering engine generates the corresponding three-dimensional audio scene in both speakers.

[0082] Fig. 7 represents only one sound source (01, C>1, r1) but the reader will understand that what follows will be repeated for each of the sound sources, and that the respective signals of the different sound sources are added together in the waves produced towards the loudspeakers (HPg, HPd).

[0083] When the binaural rendering engine operates in the frequency domain, a Fast Fourier Transform (FFT) denoted XASi(f), more simply denoted XAS in [Fig. 7], is calculated from the time pattern ASi(t). The coordinates (01, l,rl) of the sound source are applied on one hand to the HH filter] for the left ear and on the other hand to the HHr filter for the right ear.

[0084] For the left channel, the fast Fourier transform XAS of the source signal is multiplied by the filter HH] assigned the coordinates of the source, which gives the signal to be reproduced in frequency form WASg. This signal is subjected to an inverse Fourier transform calculation 21, thus obtaining the time signal WASg(t), i.e., the image of the background sound, which will be played on the left loudspeaker HPg.

[0085] For the right channel, a similar procedure is used: the fast Fourier transform of the source signal is multiplied by the filter HHr, assigned the coordinates of the source, which gives the signal to be reproduced in frequency form WASd. This signal is then subjected to an inverse Fourier transform calculation 22, thus obtaining the time-domain signal WASd(t), i.e., the background sound image, which will be played on the right loudspeaker HPd.

[0086] In an alternative implementation, when the binaural rendering engine works in the time domain, for the left channel a convolution product is performed between the sound source signal ASi(t) and the impulse response function hh / t) affected by the coordinates (0l,C>l,rl) of the sound source, thus obtaining the time signal WASg(t) which will be played on the left speaker HPg.

[0087] A similar procedure is carried out for the right channel, a convolution product is performed between the sound source signal ASi(t) and the impulse response function hhr(t) with coordinates (0l,C>l,rl) of the sound source, thus obtaining the time signal WASd(t) which will be played on the right speaker HPd.

[0088] It should be noted that to a certain extent the left and right transfer functions (HH and HHr) can be customized for the listener experiencing the three-dimensional audio scene. This includes adapting the head diameter to account for interaural delays, azimuthal angular correction (gamma correction), or combined azimuthal and elevation angular correction.

[0089] Finally, regarding [Fig. 7], if the head tracking function is present, it provides in real time a signal denoted HDT(t) representing the head movement in yaw rotation, as well as in left-right lateral tilt and front-back tilt, thus three spatial rotation coordinates. As explained elsewhere, the rendering engine compensates by injecting these values ​​with opposite signs into the sound signal generation algorithm.

[0090] Reprocessing and Improvement of Transfer Functions

[0091] We start from a set of primary (basic) auditory transfer functions, relating to the left ear and the right ear of individuals, recorded from a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) denoted ) and including amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) denoted 0j 0 j ct including amplitude and phase, a time impulse response function for the left ear (HRIRg (i,Mj)(t)) denoted #7# ) and a right-ear impulse response time function (HRIRd) (i,Mj)(t)) denoted each position Mj having polar coordinate direction 07 and ¢,,

[0092] The frequency transfer functions are in this document manipulated in digitized / numerical form, namely a sequence of values. For a large For each frequency within the human audible spectrum, the transfer function provides an amplitude and a phase. Typically, there can be 128 frequency points listed, or even 256. For each point, the amplitude and phase are encoded digitally using 8, 10, or 12 bits.

[0093] The different reprocessing steps of these raw transfer functions are illustrated in [Fig.8]. The first step, denoted S0, corresponds to making available the aforementioned raw transfer functions obtained from measurements with, where appropriate, right-left symmetrization.

[0094] It is envisaged that the spectral amplitude symmetrization step SA will further include smoothing by local average of k neighboring directions j, according to the formulation: L(«, ¢) ) + dB(wjH^~) 100951 Rie, ¢) = 0.) ) + dB^WjH^-0f ¢)) )

[0096] The notation dB represents the decimal logarithm function.

[0097] According to a more generically expressed solution, this step includes, in addition to the logarithmic transformation, where appropriate symmetrization of the spectral amplitude for all directions (0, 0) of a subject i, using an averaging function FA, according to the formulation: V($) = FA(dB, dB(H1 / -)

[0098] = FA (dB^H^^dB^

[0099] if the raw transfer functions have already been preprocessed and digitized, then the SA step is simplified and amounts to a logarithmic scaling.

[0100] According to a simple example, the averaging function FA consists of the half sum of the two values.

[0101] Then, a step denoted SB is carried out, smoothing the spectral amplitude by binaural relative deviation, normalization for a reference direction with mean on the topics, according to the wording, for a source on the left:

[0102] dB(H,(9. <f,)) ^£^(¢¢)-^(¾^)) 101031 dB(Hr(0, < / ,))= (g¢) ) +dB(Hl(e, ¢) )

[0104] The reference direction Oo^o may correspond to 00 = 15° and <F0=0. Il s’agit de filtres dit ‘image’ dans le contexte de la présente invention.

[0105] According to another solution, the reference direction Oo^o may correspond to 0O = 30° and o=O. H refers to so-called 'audio' filters in the context of the present invention.

[0106] For a right-hand source, normalization with respect to the reference direction Oo^right side can be applied in a similar manner:

[0107] dB(nr(d ¢) ) = ¢) -R1^ )

[0108] ¢) ¢)-r^o, ¢) ) +dB(H,.(e, ¢))

[0109] Next, an optional additional step, designated SB2, of flattening is carried out. of the amplitude of the filters on a CFg frontal cap according to the formulation: ( / 5( / / ,(0,0)) = for any direction 07, <t>7 included in the solid angle CFg defined by the respective bounds [0,00>] and [- <t> 0 + <t>0] left side.

[0110] According to a non-limiting example, the frontal cap CFg can correspond to the limits 0° = 30° and <b0=12o.

[0111] According to a non-limiting example, the frontal cap CFg can correspond to the limits 00 = 20° and ¢0 = 80.

[0112] It is understood that the extent of the CF frontal cap can be configured, in view of targeted applications, e.g. single source audio scene or multi source audio scene.

[0113] The above formula is given for a source located on the left. If the source is located on the right then dB (Ht for any direction 07, included in the solid angle CFd defined by the respective bounds [O,0o>] and [- Oo.+Oo] right side.

[0114] In other words, the frontal cap CFg or CFd is specific to the side of the source and confined to the side of the source, i.e. it is limited by the boundary CC at 0 = 0°.

[0115] The flattening of the filters on a front cap (CF) can be achieved according to the following alternative formulation:

[0116] ¢) ) = ^( / / ( a ¢) -L |«|. 0) ) [01171 dB(H,(0. 0) ) = 0)¢) ) +dB(Ht(e, ¢) )

[0118] It is specified here that this process flattens the HRTF on the source side, not just the left side. Therefore, the same logic must be applied symmetrically for a source located on the right.

[0119] It should also be noted that the CFg or CFd cap is not necessarily square or rectangular, it can be circular or elliptical, for example it can be delimited by the equation O2 + exc O2 < K, inside the hemisphere of the source, exc being the eccentricity of the ellipse, K being a parameter, this variant is referenced as CF' in [Fig.6].

[0120] Furthermore, a step denoted SC of symmetrization and phase averaging is carried out, according to the formulation: 101211 <h,(A0)> =3Æ"( e ' *)> + <«#((-«>* / )> )

[0122] The notation < . > denotes the phase of the complex transfer function.

[0123] The SC step can be carried out at any time relative to the other offline steps, in particular for example at the very beginning in parallel with the SA step or later.

[0124] Furthermore, a step, noted as SD, is carried out, evaluating the mean intraural delay; li(0, ¢) = argmax^ |h'g( 6, ¢) ) 101251 Ir( d, 0) = argmaxÇ |hlcl( 6 », 0) )

[0126] r(O) =77(MO)- / / (O))

[0127] fs being the sampling frequency of the HRIR impulse responses, and argmax(un) being the index of the maximum of a sequence un of digitized impulse response values, the notation * being the convolution product, w being a convolution filtering function. The convolution filtering function w can be reduced to a Dirac pulse (then we simply select the largest signal peak).

[0128] The SC and SD steps can be carried out in the order presented here, but the order on this point can be different; the SC and SD steps can be carried out before or after the other steps.

[0129] A step, denoted SE, can also be performed for interpolating spectral amplitudes, phases, and internal delays. This interpolation can consist of linear interpolation between known points. In another example, it is a second-degree polynomial interpolation between known points. The final granularity can consist of providing a value every 3°, or even every 2°, in azimuth and elevation. As will be seen later, this SE interpolation step can be performed in offline calculations or in the real-time process.

[0130] Then we proceed to a step denoted SF, of generating a set of processed transfer functions, denoted HHi and HHr, HH; ( a 0 ) = i ( ) / 2° [0!3i] HHr( 0 j = !odB(.Hr( W)

[0132] This SF step allows switching back to linear mode from logarithmic mode.

[0133] If we take the right ear as the first ear, instead of the left ear As above, this SF step is written as follows:

[0134] = j^mw) ) / 20^(,,^

[0135] But indeed this amounts to the same thing, subject to a change of sign for the angular phase shift.

[0136] Then we can proceed to a step of individualizing the internal SG delay, as a function of a head radius b of a target listener, the intraural delay being determined by with b the new head radius and a the original radius.

[0137] The radius may have been derived from measurements accompanying the transfer function measurements. The radius may have been derived from anthropological data.

[0138] The radius b can be chosen according to the listener benefiting from the audio scene rendering. For example, the radius b can be smaller than a for a listener with a small head circumference; we would refer to size XS or S.

[0139] For example, radius b may be larger than radius a for a listener with a large head circumference, we will speak of size L or XL.

[0140] For example, radius b may be larger than radius a for a listener with an average head circumference.

[0141] Then we proceed to a step denoted SH of angular correction.

[0142] According to a first simple example referred to here as 'in two dimensions', a correction is made azimuthal angular, with a parameter y that can be adapted to each user, according to the following formulation:

[0143] f(0,$) = (e + y*sin(28),^

[0144] Put another way, instead of using the azimuth angle 0 directly, we use instead a transform of the azimuth angle in the form H + y *sin ( 26 ).

[0145] The value of y is configurable.

[0146] If the 'head movement tracking' function is implemented, the SH step performs yaw movement corrections on 0 in a manner similar to that shown above.

[0147] According to another example of the SH angular correction step, azimuth and elevation are corrected according to a so-called 'three-dimensional' deformation, using f( = ( 6 + y*sin(2^), 0 + y*sin(0)). The parameters y and y' can be tailored to each user.

[0148] In an even more generic way, the step denoted SH of angular correction consists of replacing the angles 0,q> in the HRTF formulas by the angles (0', such that (0', ¢ ') = f (0, ¢) f being any azimuth and elevation correction function.

[0149] Figures 8, 9, and 10 schematically illustrate the steps described above of the proposed process, according to three hardware implementation variants. The functional distribution differs between steps performed offline and steps performed in real-time mode.

[0150] In [Fig. 8], only the SH and SR steps are performed in the target device, for example, headphones, in real time relative to the execution of the binaural rendering. The other steps were performed beforehand, by calculation, in one or more computers. It is particularly noteworthy that the SE interpolation step was performed in Upstream. It should also be noted that, as mentioned previously, some steps are optional, for example, steps SC and SG. It is understood that if step SG is not performed, the rendering is not personalized according to the target listener's head radius. Note that step SH, which corrects azimuthal angles (and elevation if applicable), can be personalized according to the target listener.

[0151] In [Fig. 9], steps SG, SH, and SR are performed in the target device, for example, headphones. Here, step SG can then be customized according to the target listener, as can step SH. Steps SA to SF are performed beforehand, by computation, in one or more computers.

[0152] In [Fig. 10], the SE, SF, SG, SH, and SR steps are performed in the target device, for example, headphones. In particular, the SE interpolation step can be performed in real time for the positions that are actually needed and used by the binaural rendering engine. This allows for a limited memory footprint and enables the calculation of relevant points at the last moment.

[0153] The SG and SH steps can be customized according to the target auditor. Only the SA to SD steps are performed upstream, by calculation, in one or more computers.

[0154] The choice between the different solutions may be dictated by the audio production chain or by the different hardware limitations in terms of computing capacity or available memory size.

[0155] Figures 2 and 3 illustrate filters normalized with respect to the reference direction 0O = 15°, this normalization being produced by the SB step described above.

[0156] On [Fig.2], curve 61 (solid line) represents the amplitude of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°.

[0157] Curve 62 (small dashed line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 63 (dashed line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 64 (long dashed line) represents the amplitude of the frequency transfer function for 0 = 45°.

[0158] Curve 65 (solid line) represents the phase of the frequency transfer function for the contralateral ear at 0°, normalized with respect to the reference direction 0° = 15°. Curve 66 (short dashed line) represents the phase of the frequency transfer function at 0° = 15°. Curve 67 (dashed line) represents the phase of the frequency transfer function at 0° = 30°. Curve 68 (long dashed line) represents the phase of the frequency transfer function at 0° = 45°.

[0159] On [Fig.3], curve 71 (solid line) represents the amplitude of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°.

[0160] Curve 72 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 73 (dashed line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 74 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.

[0161] Curve 75 (solid line) represents the phase of the frequency transfer function for the ipsilateral ear at 0°, normalized with respect to the reference direction 0° = 15°. Curve 76 (small dashed lines) represents the phase of the frequency transfer function at 0° = 15°. Curve 77 (dashed line) represents the phase of the frequency transfer function at 0° = 30°. Curve 78 represents the phase of the frequency transfer function at 0° = 45°.

[0162] We note that curve 72 is essentially flat, which confirms the effect of normalization with respect to 0=15° for the ipsilateral ear.

[0163] Figures 4 and 5 illustrate filters normalized with respect to the reference direction 00 = 30°, and flattened in a frontal cap area, these transformations being produced by the steps SB and SB2 described above.

[0164] On [Fig.4], curve 81 (solid line) represents the amplitude of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 00 = 30°.

[0165] Curve 82 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 83 (dashed line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 84 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.

[0166] Curve 85 represents the phase of the frequency transfer function for the contralateral ear at θ = 0°, normalized with respect to the reference direction θ = 30°. Curve 86 represents the phase of the frequency transfer function at θ = 15°. Curve 87 represents the phase of the frequency transfer function at θ = 30°. Curve 88 represents the phase of the frequency transfer function at θ = 45°.

[0167] On [Fig.5], curve 91 (solid line) represents the amplitude of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the reference direction 00 = 30°, with flattening of the amplitude from 0° up to 00 = 30.

[0168] Curve 92 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 93 (dashed line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 94 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.

[0169] Curve 95 (solid line) represents the phase of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the direction Reference angle 00 = 30°. Curve 96 (small dotted line) represents the phase of the frequency transfer function for 0 = 15°. Curve 97 (dashed line) represents the phase of the frequency transfer function for 0 = 30°. Curve 98 (solid line) represents the phase of the frequency transfer function for 0 = 45°.

[0170] We note that curves 91, 92, 93 are flat, which confirms the effect of flattening and normalization with respect to 30°, over the entire range from 0° to 30° (0°, 15° and 30° identical here).

[0171] It can be seen from these curves that the contralateral ear is affected in the same way in the HRTFs normalized to 15° ('image') and normalized to 30° ('audio');

[0172] Other considerations

[0173] The reprocessed filters produced by the SA to SG or SH steps constitute an improved database which can be made available to third parties for collaborations, a license to use or as part of a sale.

[0174] The inventors have noticed that other competing bases such as those of Dolby™ or similar have more chaotic amplitudes between 1kHz and 10kHz and give less good results.

[0175] Optionally, due to the symmetrization, it is possible to keep the filters in memory only for one hemisphere, right or left, and to use a planar symmetry with respect to the PMS sagittal plane performed by the real-time binaural rendering engine.

[0176] More precisely, in practice, after the phase and amplitude symmetrization steps SA and SC, data can be stored for only one hemisphere, and the complementary transformations at steps SB, SB2, and SD can be performed on only that hemisphere. The process for the other hemisphere will be reconstructed in real time by planar symmetry with respect to the sagittal plane PMS. In this way, the memory space occupied can be halved, which may be relevant for certain categories of audio hardware or processing chains.

[0177] According to another embodiment of the present invention, starting from certain HRTF filter bases, possibly already symmetrized, the flattening defined in step SB2 described above is simply applied. In this case, it may be necessary to connect the filter amplitudes to the edge of the CFg (or CFd, respectively) cap by continuity to guarantee the absence of a jump. This can be done by polynomial interpolation.

[0178] According to another optional feature, in addition to the transformations described above, a further transformation may be provided consisting of collectively boosting high-pitched sounds across all filters. Thus, for example, a boosting of sounds with frequencies above 2 or 3 kilohertz may be applied, with this boosting having an amplitude between 1 dB and 3 dB, for example.

[0179] This is illustrated in [Fig. 5], where curves 91, 92, and 93 together show an increase in amplitude above 3 kilohertz. It is noted that, due to this characteristic, the low-frequency plateau has been shown shifted below 0 dB.< / t> < / t> < / t> < / e> < / j>

Claims

Claims

1. Method for generating an audio scene in a binaural spatialization system, the method being characterized in that it comprises: S0- a step of providing a set of primary auditory transfer functions, relating to the left ear and the right ear of individuals, taken from a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) noted h' (Q. & j and comprising amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) noted H* (f). \ and comprising amplitude and phase, a time impulse response function for the left ear (HRIRg(i,Mj)(t)) noted a, ) and a time impulse response function for the right ear (HRIRd(i,Mj)(t)) noted ÿ ' each position Mj having for direction in polar coordinates Qj and <h,, le procédé comprenant : SA- une étape préalable de préparation notée SA, dans laquelle on réalise un passage en valeurs logarithmiques, avec le cas échéant symétrisation de l'amplitude spectrale pour toutes les directions ( 0, d'un sujet i, à l’aide d’une fonction de moyennage FA, selon la formulation : Lf( 0, ¢) = FA (dB (h^ , dB (H1 / -0? ÿj)) R‘ (0, ¢)= FA ( dB (h^, , dB - 0f ) la fonction dB étant la fonction logarithme décimal, SB- une étape notée SB, de lissage de l'amplitude spectrale par écart relatif binaurale, normalisation pour une direction de référence ) avec moyenne sur les sujets, selon la formulation : ¢)) = e, ÿ0) ) e, e, ¢) ) +dB(H,(s.¢)) SD- une étape notée SD, d’évaluation du retard intéraural moyen ; = ÿy E; argmax( \ ^i{0, <p) *w| ) Ir( 3, ¢) - ^E- argmax( ¢) *w| ), fs being the sampling frequency of the HRIR, and argmax(un) being the index of the maximum of a sequence un, * being the convolution product, w being a convolution filter function. SF- a step of generating a set of reprocessed transfer functions, denoted SF, HH^e, ¢) = HHr(d, ¢) = W^W) ) / 20^) SR- a step denoted SR of using the reprocessed functions 0) and HH,( B. <p) pour générer en temps réel une scène audio grâce à un moteur de rendu binaural, avec une ou plusieurs sources audios ponctuelle(s) à position(s) prescrite(s).

2. Method according to claim 1, further comprising a step denoted SB2 of flattening the amplitude of the filters on a frontal cap (CF) according to the formulation: dB(Ht(», ¢) ) = ¢)-L>(\9\. 0) ) dB(Hr( ft ¢) ¢) ) + dB(H,(e, ¢) ) for any direction Qj, ¢, included in the solid angle defined by the limits [-0O- 0] and [0o <bo]

3. The method of any one of claims 1 to 2, wherein the binaural rendering engine further comprises a listener head movement tracking function.

4. Method according to any one of claims 1 to 3, in which there is further provided, before step SF or during step SR: a step denoted SE of interpolation of the spectral amplitudes, phases and interaural delays.

5. Method according to any one of claims 1 to 4, in which the step of symmetrizing the spectral amplitude (SA) further comprises a smoothing by local average of k neighboring directions j, according to the formulation: 1! ( e, 0 ) = î E) ( dB ( WjH'g ( e „ ) ) + dB ( wfit - 0f ) ) ^dB^H^-Bf ¢,)) )

6. Method according to any one of claims 1 to 5, further comprising a step of individualizing the interaural delay (SG) according to of a head radius b of a target listener, the interaural delay being determined by èy, with b the new head radius and a the original radius.

7. Method according to any one of claims 1 to 6, further comprising a step denoted SH of angular correction, with a parameter y which can be adapted to each user, according to the formulation: / (0,0) = (3 + y^m(26), <p)

8. Method according to any one of claims 1 to 7, further comprising a step denoted SC of symmetrization and phase average, according to the formulation:<!----> )>+ ) —\ \ JJ / / the notation < . > denoting the phase of the complex transfer function.

9. Audio headset configured to implement steps SE to SR of a method according to any one of claims 1 to 8.

10. A processing system configured to implement steps S0 to SG of a method according to any one of claims 1 to 8.

Images