Method for generating an audio scene in a binaural spatialization system
By processing and reprocessing auditory transfer functions through symmetrization, smoothing, and normalization, the method addresses the limitations of individual-specific functions, achieving improved timbre reproduction and reduced comb filter effects for a broader audience in binaural spatialization systems.
Patent Information
- Application Number
- FR2023005172
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-05-25
AI Technical Summary
Existing methods for generating three-dimensional audio scenes using binaural spatialization systems fail to provide satisfactory results for a general audience due to the use of individual-specific auditory transfer functions or averages, leading to issues with timbre reproduction and comb filter effects.
A method involving a series of steps to process and reprocess auditory transfer functions, including symmetrization, smoothing, normalization, and interpolation, to create reprocessed transfer functions that are more suitable for a broader audience, ensuring accurate timbre reproduction and minimizing comb filter effects.
The reprocessed transfer functions significantly improve the reproduction of audio sources for a diverse group of users, providing a balanced and immersive three-dimensional audio experience with reduced computational load and enhanced perceived quality.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method for generating an audio scene in a binaural spatialization system
[0001] The invention relates to a method for generating an audio scene in a binaural spatialization system. The invention also relates to a system and an audio headset in which the proposed method can be implemented.
[0002] Generally speaking, it is increasingly common to need to generate a three-dimensional audio scene by synthesis, whether to reproduce stereophonic musical listening on headphones or via earplugs or in the case of immersion scenarios in a virtual reality with a virtual reality headset.
[0003] We speak of binaural hearing when both ears of an individual receive specific sound signals, corresponding to a perception of sounds in space allowing the auditory system and the brain of the individual to determine the direction of origin of a sound.
[0004] Thus, thanks to binaural hearing, spatialized sound reproduction allows a listener to perceive sound sources coming from any direction or position in space, and to have the impression of being immersed in a three-dimensional sound scene.
[0005] The signals received at the listener's right and left ears are differentiated, in particular but not exclusively, by their intensity and by the arrival and phase delay.
[0006] The term "multichannel", in processing for spatialized sound reproduction, consists of producing a representation of an acoustic scene in the form of P signals (called spatial components). These signals contain all the sounds that make up the sound scene with nevertheless weightings that depend on their direction (or "incidence") and described by P associated spatial rendering functions. It should be noted, however, that the proposed method also works for a scene with a single audio source.
[0007] The particular techniques of spatialized sound reproduction to which the present invention relates are based on the existence of acoustic transfer functions of the head between the positions in space and the auditory canal. These transfer functions, called "HRTF" (for "Head Related Transfer Functions"), concern the frequency form of the transfer functions. Their temporal form will hereinafter be designated "HRIR" (for "Head Related Impulse Response").
[0008] More particularly, we are particularly interested in a set of transfer functions hearing aids comprising for a given individual and for a plurality of measured positions M: a frequency transfer function for the left ear (HRTFg (M)(f)), a frequency transfer function for the right ear (HRTFd (M)(f)), an impulse response function for the left ear (HRIRg (M)(t)) and an impulse response function for the right ear (HRIRd (M)(t)).
[0009] There are public databases in which these auditory transfer function measurements have been listed for a plurality of individuals. In the remainder of this document, a particular individual will be assigned the index 'i' and the number of individuals will be conventionally equal to N.
[0010] The so-called ipsilateral ear corresponds to the ear that receives the sound first, and the so-called contralateral (or 'contralateral') ear corresponds to the ear that receives the sound second for the same sound source. In the remainder of the document, the left hemisphere is the half-sphere located to the left of the sagittal plane referenced by the head of the listener in an upright position, and the right hemisphere is the half-sphere located to the right of this sagittal plane.
[0011] The left ear is the ipsilateral ear for sounds originating from the left hemisphere and the right ear is the ipsilateral ear for sounds originating from the right hemisphere.
[0012] For the generation of a sound scene by synthesis in a binaural rendering engine, a convolution operation is carried out between the sound sources of the scene and the temporal impulse responses, respectively right and left, or if working in the frequency domain, a multiplication operation of the spectra then an inverse Fourier transform.
[0013] It turns out that using a set of auditory transfer functions measured on a particular individual is not satisfactory if used for a general binaural rendering engine used for a general audience.
[0014] Constructing an average of the measurements on N individuals and producing a set of average auditory transfer functions is also not entirely satisfactory.
[0015] Some have tried to work on the phase of the transfer functions as for example in the document US 10609504 but the subjective results still leave something to be desired.
[0016] The inventors sought to improve the situation, in particular by reworking the transfer function sets from public database(s), with in particular as a target a binaural rendering in an audio headset. Particular attention is paid to the restitution of the timbre of the audio sources, and to the elimination of possible comb filter effects.
[0017] The method and the proposed system can also be used in a so-called “transaural” context with listening on loudspeakers.
[0018] The inventors also kept as their objective to be able to offer a possibility of individualization with respect to the listener who will be immersed in the three-dimensional audio scene.
[0019] For this purpose, a method is therefore proposed here for generating an audio scene in a binaural spatialization system, the method being characterized in that it comprises: S0- a step of providing a set of primary (basic) auditory transfer functions, relating to the left ear and the right ear of individuals, taken from a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) noted $ ) Ct including amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) noted H6j (fi ) ct including amplitude and phase, a time-domain impulse response function for the left ear (HRIRg (i,Mj)(t)) noted and a time impulse response function for the right ear (HRIRd (i,Mj)(t)) denoted each position Mj having the direction in polar coordinates 07 and ¢, (azimuth and elevation), the method comprising: SA- a preliminary preparation step noted SA, in which a passage in logarithmic values is carried out, with where appropriate (if this has not already been done in the public base) a symmetrization of the spectral amplitude for all directions (d (fi ) of a subject i, using an averaging function FA, according to the formulation: -FA dB^-O^))
[0020] R'(0^) = FA [dB (h^, dB (H^- 0^ (fi^)
[0021] the dB function being the decimal logarithm function, the FA function being able to be in a simplified case the half sum, SB a step noted SB, of smoothing of the spectral amplitude by binaural relative deviation, normalization for a reference direction with average over the subjects, according to the formulation:
[0022] =
[0025] dB(H r(B, ¢)) = B, 0. ¢)) + dB(Hl(0, (¢))
[0024] SD- a step noted SD, for evaluating the average interaural delay; = argmax ( |Ag(0, ¢) *vv| )
[0025] — ±argmax( |hld*w| )
[0026] t{ 0, U
[0027] fs being the sampling frequency of the HRIR impulse responses, and argmax(un) being the index of the maximum of a sequence u», of the digitized values of the impulse response, the notation * being the convolution product, w being a convolution filter function, which can be reduced to a Dirac impulse (then we simply select the largest signal peak),
[0028] SF- a step noted SF, of generating a set of reprocessed transfer functions, noted HHi and HHr, HH^O, ¢) = V20
[0029] HHr{ 0^) =
[0030] SR- a step noted SR of using the reprocessed functions HH[(0, and HHr( 0, ( / >) to generate in real time an audio scene using a binaural rendering engine, with one or more point audio sources at prescribed position(s).
[0031] Thanks to the provisions presented above, the development and use of the reprocessed transfer functions gives very satisfactory results. The timbre of the audio source or audio sources is restored in a very satisfactory manner for a set of users in a tested panel.
[0032] It is further noted that the normalization of the amplitudes of spectral filters, with respect to a reference direction j, proves to be particularly beneficial, notably from the point of view of general balancing.
[0033] Advantageously, it is noted that the comb filter effects are erased.
[0034] The average effect in relation to all the individuals measured makes it possible to work on an average individual and to maximize the suitability of the reworked transfer functions in relation to the general population capable of using devices implementing the proposed method.
[0035] It is noted that the number of audio sources participating in the generation of synthesized audio signals is arbitrary. A single audio source may correspond to a case of a video conference with a person who delivers a speech in a place without any other sound source in the audio scene. In other configurations, a multiple sound environment is provided with multiple sound sources which are located in front of and behind the listeners as in the case of immersion in a subjective view video game (for example of the FPS type for 'First Person Shoot') or as in the case of simulators of all types.
[0036] It should also be noted that according to the proposed method, we switch to logarithmic values to perform calculations and then conversely we switch back to linear values before injecting the filters with their amplitudes and phases into the binaural rendering engine. This allows subtractions to be made instead of divisions and this lightens the overall computational load.
[0037] As will be specified later, certain steps are carried out in a so-called offline mode, i.e. upstream of actual use in a rendering engine. binaural in an audio playback device, while conversely some steps are carried out in real time, namely at the time of the generation of sound waves in an audio playback device. As we will see, there are several possible solutions for the functional division between offline steps and real-time steps.
[0038] According to one aspect, the method may further comprise a step denoted SB2 flattening the amplitude of filters on a frontal cap (CFg) for the ear ipsilateral, according to the formulation: for any direction 07, <h, comprise dans the solid angle defined by the respective limits [O,0o>] and [- <b0 +<b0] côté gauche. La formula is given for a source located on the left. If the source is located on the right then dB(Hr( dB(Hr( da 0()) ) for any direction 07, <h, comprise in the solid angle CFd defined by the respective limits [O,0o>] and [- <b0 + 0] right side.
[0039] Advantageously, the amplitudes applied for the flattening correspond to the amplitudes of the filter of the reference direction OqOq of the ipsilateral side. According to one option, the filters for the contralateral ear remain unchanged, in particular concerning the amplitudes and the minimum phases.
[0040] Whereby a flat filter is applied without correction, on the amplitude of the filter of the ipsilateral ear, in a frontal cap. The test campaigns carried out by the inventors revealed that this arrangement was favorable from the point of view of perceived quality on the part of the users of the test panel.
[0041] We note that, at the boundaries of the cap, there is no jump or discontinuity, because the amplitudes have previously been normalized with respect to the reference amplitude in the direction [OoOoL The filter is continuous and derivable over the entire range of angles.
[0042] According to another formulation of this step, the step noted SB2 of flattening the amplitude of the filters on a frontal cap (CF), can be carried out according to the formulation: 100431 dB(H,(". 0) ) = ¢) - A l»|. *)) dB(Hr(e, (R\e, ¢)¢) )
[0044] It is specified here that this step of the process flattens the HRTF on the side of the source, here the left one. Therefore the same logic is to be applied symmetrically for a source located on the right.
[0045] According to one aspect, the binaural rendering engine further comprises a head tracking function. In practice, a 3-axis gyroscopic sensor provides the yaw, roll and pitch movements to an audio headset on which the binaural rendering engine is implemented. It is noted here that the gyroscopic sensor can be integrated into the audio headset. Alternatively, the yaw movement can be determined by the use of one or more cameras, whether or not they are embedded in the headset. Thus, the rendering engine can compensate in real time for the listener's head movements by calling the corresponding inverse positions in the database: e.g., if a listening subject turns his head 15° to his left, a sound initially placed at 0° in front of him will shift to his right by 15°.
[0046] According to one aspect, the method may further comprise, before step SF or during step SR, a step denoted SE of interpolation of the spectral amplitudes, phases and interaural delays.
[0047] By means of which, starting from a base of known transfer function sets with a fairly large step, for example every 15° or every 10°, it is possible to provide, thanks to interpolation, a base of transfer function sets of better granularity, for example every 3° or even every 2°.
[0048] It should be noted that this interpolation step can be carried out in the offline steps or can be carried out in real time with the binaural rendering engine.
[0049] According to one aspect, the step of symmetrization of the spectral amplitude (SA) may further comprise smoothing by local average of k neighboring directions j, according to the formulation: L\9, #)-^(^(^(^ ef ) R* (ft #) = 4 EJ (dB ((Bj, ¢.)) + dB (j))
[0050] The filters are thus smoothed not only on individuals, but also in space.
[0051] According to one aspect, the method may further comprise a step of individualizing the interaural delay (SG) as a function of a head radius b of a target listener, the interaural delay being determined by with b the new head radius and a the original radius, i.e. the reference head radius.
[0052] According to one aspect, the method may further comprise a step denoted SH of angular correction, in fact an azimuthal angular correction, with a parameter y which can be adapted to each user, according to the formulation:
[0053] / (0,0) = (0 + y*sin(20),0)
[0054] The parameter y is preferably configurable, and can even be individualized depending on the listener benefiting from the binaural audio rendering.
[0055] In other words, instead of using the azimuth angle 3 directly, we use instead a transform of the azimuth angle in the form 0 + y*sin(20) • H turns out that the qualitative result is further improved by this angular deformation.
[0056] According to a related aspect, the method may comprise a step denoted SH of angular correction with an angular correction in azimuth and elevation where the angles 0,q> in the HRTF formulas are replaced by the angles (0',q / ) such that (0',q / ) = f (0,q>).
[0057] According to an example, we can choose (0', 0') = f(0, 0) = (0 + y*sin(20), 0 + y'*sin(0)).
[0058] The parameters y and y' can be adapted to each user.
[0059] In other words, instead of using the azimuth angle 8 directly, we use instead a transform of the azimuth angle in the form 0 + y*sin(20) and instead of using the elevation angle q> directly, we use instead a transform of the elevation angle in the form 0 + y'*sin( 0) •
[0060] According to one aspect, the method may further comprise a step denoted SC of symmetrization and averaging of the phase, according to the formulation: <W)> = + ) 100611 )
[0062] The notation < . > denotes the phase of the complex transfer function.
[0063] This further improves the averaging function of the individuals who participated to transfer function measurement campaigns.
[0064] The invention also relates to an audio headset configured to implement steps SE to SR of a method as described previously. Depending on the configurations, the audio headset can carry out steps SG to SR or SH to SR.
[0065] The invention also relates to a processing system configured to implement steps S0 to SG of a method as described previously.
[0066] According to another embodiment of the present invention, starting from certain bases of HRTF filters, possibly already symmetrized, the flattening defined in step SB2 is simply applied. In this case, it may be necessary to connect the filters by continuity to the edge of the cap CFg, (respectively CFd) to guarantee the absence of jumps. This can be done by a polynomial interpolation.
[0067] The invention will be further detailed by the description of non-limiting embodiments, and on the basis of the appended figures illustrating variants of the invention, in which: - [Fig.l] is a schematic diagram illustrating the spherical coordinates of points and directions in space around an individual, for example a listener in the context of listening or in the context of measuring auditory transfer functions HRTF; - [Fig.2] illustrates an example of frequency transfer function amplitude, with the addition of the phase, for the contralateral ear with normalization relative to the 15° direction; - [Fig.3] illustrates an example of frequency transfer function amplitude, with additionally the phase for the ipsilateral ear with normalization relative to the 15° direction; - [Fig.4] illustrates an example of frequency transfer function amplitude, with additionally the phase for the contralateral ear with normalization relative to the 30° direction; - [Fig.5] illustrates an example of frequency transfer function amplitude, with plus phase, with plus phase for the ipsilateral ear with normalization relative to the 30° direction, and flattening on the frontal cap; - [Fig.6] is analogous to [Fig.l] and illustrates in particular areas of the cap frontal where the filtering spectra are flattened; - [Fig.7] schematically illustrates an audio scene restitution using a binaural rendering engine, for a single ASi audio source; - [Fig.8] schematically illustrates an example of a succession of steps of the proposed method according to a first variant of hardware implementation; - [Fig.9] schematically illustrates an example of a succession of stages of the method proposed according to a second hardware implementation variant; - [Fig. 10] schematically illustrates an example of a succession of steps of the proposed method according to a third variant of hardware implementation.
[0068] In the different figures, the same references designate identical or similar elements. For reasons of clarity of the presentation, certain elements are not necessarily represented to scale.
[0069] Generalities and system
[0070] Generally, a spatialized binaural sound reproduction allows a listener U to be immersed in an audio scene and to perceive sound sources coming from different directions and positions in space, while the listener U only has two loudspeakers HPg and HPd.
[0071] In the example of [Fig.7], the restitution organ is an audio headset 2 with the left loudspeaker HPg arranged opposite the left ear OG of the listener and the right loudspeaker HPd arranged opposite the right ear OD of the listener U.
[0072] In other configurations, the restitution organ can be a pair of in-ear earphones such as for example “earplugs”.
[0073] In yet another configuration, in a so-called open-air listening context sometimes called 'transaural™', two restitution devices are provided, such as conventional loudspeakers, with cross-talk cancellation. The proposed invention can also be applied in this transaural configuration, with two or more loudspeakers.
[0074] For the different configurations mentioned above, the two loudspeakers can be controlled by a single control unit directly in wired mode, or the loudspeakers can be controlled by a single control unit but via a Bluetooth type wireless connection.
[0075] In one case, the audio scene to be rendered may consist of a single sound source, denoted ASi in [Fig.7]. In another case, the audio scene to be rendered consists of a plurality of sound sources ASj. For example, the audio scene to be rendered may comprise NbS sound sources.
[0076] Each sound source to be reproduced, such as for example the first source ASi, is located at a position Ml characterized by its polar coordinates 01 and ¢1 and its distance rl, as illustrated in [Fig.l]. The polar coordinates will be used throughout this document. The first coordinate 0 represents the azimuth relative to a “straight ahead” direction, with positive angles being marked to the right or left depending on the side of the source concerned. The second coordinate (also noted <j>) corresponds to the elevation also called “site” relative to a horizontal direction, the positive angles being located upwards.
[0077] In [Fig.l], the plane marked PH is the horizontal reference plane for the listener with a straight head. The plane marked PMS is the mid-sagittal plane, it separates the left hemisphere from the right hemisphere. The plane marked PVI is the interaural vertical plane (it passes through 0=90°). When the system is equipped with the head movement tracking function, the deviation of the actual position from the straight head position is compensated for by the same angle values by assigning them an opposite sign. For the listener, the sound source always appears to be at its given position in space, regardless of the small head movements that the listener may make.
[0078] The reproduction of an audio scene is based on the use of acoustic transfer functions of the head between the positions in space (0,0) and the auditory canal, called auditory transfer functions. These transfer functions, called "HRTF" (for "Head Related Transfer Functions"), concern the frequency form of the transfer functions. Their temporal form, i.e. the impulse response, will hereinafter be referred to as "HRIR" (for "Head Related Impulse Response").
[0079] In the context of binaural restitution, we then use the transfer functions following: a frequency transfer function for the left ear (HRTFg(M)(f)), a frequency transfer function for the right ear (HRTFd(M)(f)), an impulse response function for the left ear (HRIRg(M)(t)) and an impulse response function for the right ear (HRIRd(M)(t)), M being the position of the source in polar coordinates, f the frequency and t the time.
[0080] Advantageously according to the present invention, instead of using standard transfer functions, after having modified and improved these transfer functions, reprocessed transfer functions are used, denoted in the present document HHi and HHr. Their frequency form will be denoted HH] and HHr and their time form will be denoted hh] and hhr. The way of obtaining these reprocessed transfer functions will be the subject of detailed explanations in the remainder of this document.
[0081] Returning to [Fig.7], the restitution of the audio scene is carried out by a binaural rendering engine. The binaural rendering engine has at its disposal the reprocessed transfer functions in their frequency form or in their time form. Thanks to these functions, for each sound source ASi of the audio scene to be reproduced, characterized by the wave temporal pattern ASi(t) of the sound source as well as its polar coordinates (0i, <e>i) and its distance to the listener ri, the binaural rendering engine generates the corresponding three-dimensional audio scene in both speakers.
[0082] [Fig.7] only represents one sound source (01, C>1, r 1) but the reader will understand that what follows will be repeated for each of the sound sources, and that the respective signals of the different sound sources are added to the waves produced towards the loudspeakers (HPg, HPd).
[0083] When the binaural rendering engine works in the frequency domain, from the time pattern ASi(t) we calculate a fast Fourier transform (FFT) noted XASi(f), more simply noted XAS in [Fig.7]. The coordinates (01, l,rl) of the sound source are applied on the one hand to the HH filter] for the left ear and on the other hand to the HHr filter for the right ear.
[0084] For the left channel, the fast Fourier transform XAS of the source signal is multiplied by the filter HH] assigned the source coordinates, which gives the signal to be reproduced in frequency form WASg. This signal is subject to an inverse Fourier transform calculation 21 and we thus obtain the time signal WASg(t), i.e. the sound background image, which will be played on the left loudspeaker HPg.
[0085] For the right channel, we proceed in a similar manner, we multiply the fast Fourier transform of the source signal by the filter HHr assigned the coordinates of the source which gives the signal to be reproduced in frequency form WASd. This signal is the subject of an inverse Fourier transform calculation 22 and we thus obtain the time signal WASd(t), i.e. sound background image, which will be played on the right loudspeaker HPd.
[0086] In an alternative implementation, when the binaural rendering engine works in the time domain, for the left channel we carry out a convolution product between the signal of the sound source ASi(t) and the impulse response function hh / t) assigned the coordinates (0l,C>l,rl) of the sound source, we thus obtain the time signal WASg(t) which will be played on the left loudspeaker HPg.
[0087] We proceed in a similar manner for the right channel, we carry out a convolution product between the signal of the sound source ASi(t) and the impulse response function hhr(t) assigned the coordinates (0l,C>l,rl) of the sound source, we thus obtain the time signal WASd(t) which will be played on the right loudspeaker HPd.
[0088] It should be noted that to some extent the left and right transfer functions (HH] and HHr) can be personalized with respect to the listener who will experience the three-dimensional audio scene. This concerns the adaptation of the head diameter with regard to interaural delays, the azimuthal angular correction ('gamma' correction) or the combined azimuthal and elevation angular correction.
[0089] To finish on [Fig.7], if the head tracking function is present, it provides in real time a signal noted HDT(t) representative of the movement of the head in yaw rotation, as well as in left-right lateral tilting and in front-back tilting, therefore three spatial rotation coordinates. As explained elsewhere, the rendering engine compensates by injecting these values with an opposite sign into the sound signal generation algorithm.
[0090] Reprocessing and Improvement of Transfer Functions
[0091] We start from a set of primary (basic) auditory transfer functions, relating to the left ear and the right ear of individuals, taken from a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) noted ) and including amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) noted 0j 0 j ct including amplitude and phase, a time impulse response function for the left ear (HRIRg (i,Mj)(t)) denoted #7 # ) and a time-domain impulse response function for the right ear (HRIRd (i,Mj)(t)) noted each position Mj having the direction in polar coordinates 07 and ¢,,
[0092] The frequency transfer functions are in this document manipulated in the digitized / digital form, namely a sequence of values. For a large number of frequencies located in the human audible spectrum, the transfer function gives an amplitude and a phase. There can typically be 128 frequency points listed or even 256 frequency points listed. For each of the points, the amplitude and the phase are encoded in digital format on 8, 10 or 12 bits.
[0093] The different stages of reprocessing these raw transfer functions are illustrated in [Fig.8]. The first stage noted S0 corresponds to the provision of the raw transfer functions mentioned above from measurements with right-left symmetrization where appropriate.
[0094] It is expected that the step of symmetrization of the spectral amplitude SA further comprises a smoothing by local average of k neighboring directions j, according to the formulation: L(«, ¢) ) + dB(wjH^~) 100951 Rie, ¢) = 0.) ) + dB^WjH^-0f ¢)) )
[0096] The notation dB represents the decimal logarithm function.
[0097] According to a solution expressed in a more generic way, this step includes, in addition to the transition to logarithm, where appropriate symmetrization of the spectral amplitude for all directions (0. 0) of a subject i, using an averaging function FA, according to the formulation: V($) = FA(dB, dB(H1 / -)
[0098] = FA (dB^H^^dB^
[0099] if the raw transfer functions have already been preprocessed and digitized, then the SA step is simplified and comes down to a transition to a logarithmic scale.
[0100] According to a simple example, the averaging function FA consists of the half-sum of the two values.
[0101] Then we proceed to a step noted SB, of smoothing the spectral amplitude by binaural relative deviation, normalization for a reference direction with average on the subjects, according to the formulation, for a source on the left:
[0102] dB(H,(9. <f,)) ^£^(¢¢)-^(¾^)) 101031 dB(Hr(0, < / ,))= (g¢) ) +dB(Hl(e, ¢) )
[0104] The reference direction Oo^o can correspond to 0O = 15° and <F0=0. Il s’agit de filtres dit ‘image’ dans le contexte de la présente invention.
[0105] According to another solution, the reference direction Oo^o can correspond to 0O = 30° and o=O. These are so-called 'audio' filters in the context of the present invention.
[0106] For a right-hand source, we can similarly apply the normalization with respect to the reference direction Oo^o right side:
[0107] dB(nr(d ¢) ) = ¢) -R1^ )
[0108] ¢) ¢)-r^o, ¢) ) +dB(H,.(e, ¢))
[0109] Then, we optionally carry out a step denoted SB2 of flattening. of the amplitude of the filters on a CFg frontal cap according to the formulation: ( / 5( / / ,(0,0)) = for any direction 07, <t>7 included in the solid angle CFg defined by the respective limits [O,0o>] and [- <t> 0 + <t>0] left side.
[0110] According to a non-limiting example, the frontal cap CFg can correspond to the terminals 0o = 30° and <b0=12o.
[0111] According to a non-limiting example, the frontal cap CFg can correspond to the terminals 0O = 20° and ¢0 = 80.
[0112] It is understood that the extent of the frontal cap CF can be configurable, with a view to targeted applications, e.g. single-source audio scene or multi-source audio scene.
[0113] The above formula is given for a source located on the left. If the source is located on the right then dB (Ht for any direction 07, included in the solid angle CFd defined by the respective terminals [O,0o>] and [- Oo.+Oo] on the right side.
[0114] In other words, the frontal cap CFg or CFd is specific to the side of the source and confined to the side of the source, i.e. it is limited by the boundary CC at 0 = 0°.
[0115] The flattening of the filters on a frontal cap (CF) can be carried out according to the formulation otherwise written:
[0116] ¢) ) = ^( / / ( a ¢) -L |«|. 0) ) [01171 dB(H,(0. 0) ) = 0)¢) ) +dB(Ht(e, ¢) )
[0118] It is specified here that this process flattens the HRTF on the source side, not just the left one. Therefore the same logic is to be applied symmetrically for a source located on the right.
[0119] It should also be noted that the cap CFg or CFd is not necessarily square or rectangular, it can be circular or elliptical, for example it can be delimited by the equation 02 + exc O2 < K, inside the hemisphere of the source, exc being the eccentricity of the ellipse, K being a parameter, this variant is reference CF' in [Fig.6].
[0120] Furthermore, we proceed to a step noted SC of symmetrization and average of the phase, according to the formulation: 101211 <h,(A0)> =3Æ"( e ' *)> + <«#((-«>* / )> )
[0122] The notation < . > denotes the phase of the complex transfer function.
[0123] Step SC can be performed at any time relative to the other offline steps, in particular for example at the very beginning in parallel with step SA or later.
[0124] Furthermore, we proceed to a step noted SD, of evaluation of the average interaural delay; li(0, ¢) = argmax^ |h'g( 6, ¢) ) 101251 Ir( d, 0) = argmaxÇ |hlcl( 6», 0) )
[0126] r(O) =77(MO)- / / (O))
[0127] fs being the sampling frequency of the HRIR impulse responses, and argmax(un) being the index of the maximum of a sequence un, of the digitized values of the impulse response, the notation * being the convolution product, w being a convolution filter function. The convolution filter function w can be reduced to a Dirac impulse (then we simply select the largest signal peak).
[0128] Steps SC and SD may be performed in the order they are presented here, but the order on this point may be different, steps SC and SD may be performed before or after the other steps.
[0129] We can also carry out a step denoted SE of interpolation of the spectral amplitudes, phases and interaural delays. This interpolation can consist of a linear interpolation between the known points. According to another example, it is a second-degree polynomial interpolation between the known points. The final granularity can consist of providing a value every 3°, or even a value every 2°, in azimuth and elevation. As will be seen later, this SE interpolation step can be carried out in offline calculations or in the real-time process.
[0130] Then we proceed to a step noted SF, of generating a set of reprocessed transfer functions, noted HHi and HHr, HH; ( a 0 ) = i ( ) / 2° [0!3i] HHr( 0 j = !odB(.Hr( W)
[0132] This SF step allows you to return to linear mode from logarithmic mode.
[0133] If we take the right ear as the first ear, instead of the left ear as above, then this SF step is written as follows:
[0134] = j^mw) ) / 20^(,,^
[0135] But in fact it comes to the same thing, with a change of sign for the angular phase shift.
[0136] Then we can proceed to a step of individualization of the interaural delay called SG, as a function of a head radius b of a target listener, the interaural delay being determined by with b the new head radius and a the original radius.
[0137] The radius a may come from measurements accompanying the transfer function measurements. The radius a may come from anthropological data.
[0138] The radius b can be chosen according to the listener benefiting from the audio scene rendering. For example, the radius b can be smaller than a for a listener with a small head circumference; we will speak of size XS or S.
[0139] For example, radius b may be larger than radius a for a listener with a large head circumference, we will speak of size L or XL.
[0140] For example, radius b may be larger than radius a for a listener with an average head circumference.
[0141] Then we proceed to a step noted SH of angular correction.
[0142] According to a first simple example called here 'in two dimensions' we make a correction azimuthal angular, with a parameter y that can be adapted to each user, according to the formulation:
[0143] f(0,$) = (e + y*sin(28),^
[0144] In other words, instead of using the azimuth angle 0 directly, we use instead a transform of the azimuth angle in the form H + y *sin ( 26 ).
[0145] The value of y is configurable.
[0146] If the 'head tracking' function is implemented, the SH step performs the yaw motion corrections on 0 in a similar manner to what was presented above.
[0147] According to another example of the SH step of angular correction, we correct in azimuth and in elevation, according to a so-called 'three-dimensional' deformation, we use f ( = ( 6 + y*sin(2^), 0 + y*sin(0)) . The parameters y and y' can be adapted to each user.
[0148] In an even more generic way, the step noted SH of angular correction consists of replacing the angles 0,q> in the HRTF formulas by the angles (0', such that (0', ¢ ') = f (0, ¢) f being any function of correction in azimuth and elevation.
[0149] Figures 8, 9 and 10 schematically illustrate the steps described above of the proposed method, according to three hardware implementation variants. The functional distribution differs between steps carried out in offline mode and steps carried out in real-time mode.
[0150] In [Fig.8], only the SH and SR steps are performed in the target device, for example an audio headset, in real time relative to the execution of the binaural rendering. The other steps were performed upstream, by calculation, in one or more computers. It is noted in particular that the SE interpolation step was performed in upstream. It should also be noted that, as mentioned previously, certain steps are optional, for example the SC and SG steps. It is understood that if the SG step is not carried out, the rendering is not personalized according to the head radius of the target listener. It should be noted that the SH step of azimuthal angular correction (and where applicable in elevation) can be personalized according to the target listener.
[0151] In [Fig.9] t the steps SG, SH and SR are carried out in the target device, for example an audio headset. Here the step SG can then be personalized according to the target listener, just like the step SH. The steps SA to SF are carried out upstream, by calculation, in one or more computers.
[0152] In [Fig. 10] the steps SE, SF, SG, SH and SR are performed in the target device, for example an audio headset. In particular, the interpolation step SE can be performed in real time for the positions that are actually needed and used by the binaural rendering engine. This allows for a limited memory space and for calculating the interesting points at the last moment.
[0153] Steps SG and SH can be customized according to the target listener. Only steps SA to SD are performed upstream, by calculation, in one or more computers.
[0154] The choice between the different solutions can be dictated by the audio production chain or by the different hardware limitations in terms of computing capacity or available memory size.
[0155] Figures 2 and 3 illustrate filters normalized with respect to the reference direction 0O = 15°, this normalization being produced by the step SB described above.
[0156] In [Fig.2], curve 61 (solid line) represents the amplitude of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°.
[0157] Curve 62 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 63 (dash line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 64 (long dotted line) represents the amplitude of the frequency transfer function for 0 = 45°.
[0158] Curve 65 (solid line) represents the phase of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°. Curve 66 (small dotted line) represents the phase of the frequency transfer function for 0 = 15°. Curve 67 (dash line) represents the phase of the frequency transfer function for 0 = 30°. Curve 68 (long dotted line) represents the phase of the frequency transfer function for 0 = 45°.
[0159] In [Fig.3], curve 71 (solid line) represents the amplitude of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°.
[0160] Curve 72 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 73 (dash line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 74 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.
[0161] Curve 75 (solid line) represents the phase of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 15°. Curve 76 (small dotted line) represents the phase of the frequency transfer function for 0 = 15°. Curve 77 (dash line) represents the phase of the frequency transfer function for 0 = 30°. Curve 78 represents the phase of the frequency transfer function for 0 = 45°.
[0162] We note that curve 72 is essentially flat, which confirms the effect of normalization with respect to 0=15° for the ipsilateral ear.
[0163] Figures 4 and 5 illustrate filters normalized with respect to the reference direction 0O = 30°, and flattened in a frontal cap area, these transformations being produced by steps SB and SB2 described above.
[0164] In [Fig.4], curve 81 (solid line) represents the amplitude of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 30°.
[0165] Curve 82 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 83 (dash line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 84 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.
[0166] Curve 85 represents the phase of the frequency transfer function, for the contralateral ear, for 0 = 0°, normalized with respect to the reference direction 0 o = 30°. Curve 86 represents the phase of the frequency transfer function for 0 = 15°. Curve 87 represents the phase of the frequency transfer function for 0 = 30°. Curve 88 represents the phase of the frequency transfer function for 0 = 45°.
[0167] In [Fig.5], curve 91 (solid line) represents the amplitude of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the reference direction 0O = 30°, with flattening of the amplitude from 0° to 0O = 30.
[0168] Curve 92 (small dotted line) represents the amplitude of the frequency transfer function for 0 = 15°. Curve 93 (dash line) represents the amplitude of the frequency transfer function for 0 = 30°. Curve 94 (solid line) represents the amplitude of the frequency transfer function for 0 = 45°.
[0169] Curve 95 (solid line) represents the phase of the frequency transfer function, for the ipsilateral ear, for 0 = 0°, normalized with respect to the direction reference 0O = 30°. Curve 96 (small dotted line) represents the phase of the frequency transfer function for 0 = 15°. Curve 97 (dash line) represents the phase of the frequency transfer function for 0 = 30°. Curve 98 (solid line) represents the phase of the frequency transfer function for 0 = 45°.
[0170] We note that curves 91,92,93 are flat, which confirms the effect of flattening and normalization relative to 30°, over the entire range from 0° to 30° (0°, 15° and 30° identical here).
[0171] It can be seen from these curves that the contralateral ear is affected in the same way in the HRTFs normalized to 15° ('image') and normalized to 30° ('audio');
[0172] Other considerations
[0173] The reprocessed filters produced by steps SA to SG or SH constitute an improved database that can be made available to third parties for collaborations, a license of use or as part of a sale.
[0174] The inventors have noticed that other competing bases such as those from Dolby™ or similar have more choppy amplitudes between 1kHz and 10kHz and give less good results.
[0175] Optionally, due to the symmetrization, it is possible to keep in memory the filters only for one hemisphere, right or left, and to use a plane symmetry with respect to the PMS sagittal plane produced by the real-time binaural rendering engine.
[0176] More precisely, in practice, after the symmetrization steps of the phase and the amplitudes SA and SC, we can only keep data for one hemisphere and make the complementary transformations at steps SB, SB2, SD only on one hemisphere. The process for the other hemisphere will be reconstructed in real time by plane symmetry with respect to the sagittal plane PMS. In this way, we can occupy a memory space divided by 2, which can be relevant for certain categories of hardware or audio processing chains.
[0177] According to another embodiment of the present invention, starting from certain HRTF filter bases, possibly already symmetrized, the flattening defined in step SB2 explained above is simply applied. In this case, it may be necessary to connect by continuity the amplitudes of the filters to the edge of the cap CFg, (respectively CFd) to guarantee the absence of jumps. This can be done by a polynomial interpolation.
[0178] According to another optional characteristic, in addition to the transformations described above, a complementary transformation can be provided consisting of raising the high-pitched sounds collectively on all the filters. Thus, for example, a reinforcement of sounds with a frequency higher than 2 or 3 kilohertz can be practiced, this reinforcement being of an amplitude between 1 dB and 3 dB for example.
[0179] This is illustrated in [Fig.5], where curves 91, 92, 93 together show an increase in amplitude above 3 kilohertz. Note that because of this characteristic, the low-frequency plateau has been shown shifted, below 0 dB.< / t> < / t> < / t> < / e> < / j>
Claims
Claims
1. Method for generating an audio scene in a binaural spatialization system, the method being characterized in that it comprises: S0- a step of providing a set of primary auditory transfer functions, relating to the left ear and the right ear of individuals, taken from a set of N individuals, the set of primary auditory transfer functions comprising for each individual i and for each of a plurality of measured positions Mj: a frequency transfer function for the left ear (HRTFg(i,Mj)(f)) noted h' (Q. & j and comprising amplitude and phase, a frequency transfer function for the right ear (HRTFd(i,Mj)(f)) noted H* (f). \ and comprising amplitude and phase, a time impulse response function for the left ear (HRIRg(i,Mj)(t)) noted a, ) and a time impulse response function for the right ear (HRIRd(i,Mj)(t)) noted ÿ ' each position Mj having for direction in polar coordinates Qj and <h,, le procédé comprenant : SA- une étape préalable de préparation notée SA, dans laquelle on réalise un passage en valeurs logarithmiques, avec le cas échéant symétrisation de l'amplitude spectrale pour toutes les directions ( 0, d'un sujet i, à l’aide d’une fonction de moyennage FA, selon la formulation : Lf( 0, ¢) = FA (dB (h^ , dB (H1 / -0? ÿj)) R‘ (0, ¢)= FA ( dB (h^, , dB - 0f ) la fonction dB étant la fonction logarithme décimal, SB- une étape notée SB, de lissage de l'amplitude spectrale par écart relatif binaurale, normalisation pour une direction de référence ) avec moyenne sur les sujets, selon la formulation : ¢)) = e, ÿ0) ) e, e, ¢) ) +dB(H,(s.¢)) SD- une étape notée SD, d’évaluation du retard intéraural moyen ; = ÿy E; argmax( \ ^i{0, <p) *w| ) Ir( 3, ¢) - ^E- argmax( ¢) *w| ), fs being the sampling frequency of the HRIR, and argmax(un) being the index of the maximum of a sequence un, * being the convolution product, w being a convolution filter function. SF- a step of generating a set of reprocessed transfer functions, denoted SF, HH^e, ¢) = HHr(d, ¢) = W^W) ) / 20^) SR- a step denoted SR of using the reprocessed functions 0) and HH,( B. <p) pour générer en temps réel une scène audio grâce à un moteur de rendu binaural, avec une ou plusieurs sources audios ponctuelle(s) à position(s) prescrite(s).
2. Method according to claim 1, further comprising a step denoted SB2 of flattening the amplitude of the filters on a frontal cap (CF) according to the formulation: dB(Ht(», ¢) ) = ¢)-L>(\9\. 0) ) dB(Hr( ft ¢) ¢) ) + dB(H,(e, ¢) ) for any direction Qj, ¢, included in the solid angle defined by the limits [-0O- 0] and [0o <bo]
3. The method of any one of claims 1 to 2, wherein the binaural rendering engine further comprises a listener head movement tracking function.
4. Method according to any one of claims 1 to 3, in which there is further provided, before step SF or during step SR: a step denoted SE of interpolation of the spectral amplitudes, phases and interaural delays.
5. Method according to any one of claims 1 to 4, in which the step of symmetrizing the spectral amplitude (SA) further comprises a smoothing by local average of k neighboring directions j, according to the formulation: 1! ( e, 0 ) = î E) ( dB ( WjH'g ( e „ ) ) + dB ( wfit - 0f ) ) ^dB^H^-Bf ¢,)) )
6. Method according to any one of claims 1 to 5, further comprising a step of individualizing the interaural delay (SG) according to of a head radius b of a target listener, the interaural delay being determined by èy, with b the new head radius and a the original radius.
7. Method according to any one of claims 1 to 6, further comprising a step denoted SH of angular correction, with a parameter y which can be adapted to each user, according to the formulation: / (0,0) = (3 + y^m(26), <p)
8. Method according to any one of claims 1 to 7, further comprising a step denoted SC of symmetrization and phase average, according to the formulation:<!----> )>+ ) —\ \ JJ / / the notation < . > denoting the phase of the complex transfer function.
9. Audio headset configured to implement steps SE to SR of a method according to any one of claims 1 to 8.
10. A processing system configured to implement steps S0 to SG of a method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Audio signal processing method and apparatus for binaural rendering using phase response characteristics
US10609504B2
Binaural synthesis, head-related transfer functions, and uses thereof
EP0746960B1
HRTF personalization based on anthropometric features
US20150312694A1
Individualization of head related transfer functions for presentation of audio content
US20210112364A1
