Object-based audio spatializer using crosstalk equalization

Abstract

To improve the quality and accuracy of virtual 3D object placement in virtual sound generation systems for video games and other applications. [Solution] The 3D sound spatializer provides delay-compensated HRTF interpolation techniques, efficient crossfading between current and delayed HRTF filter results, and object-by-object equalization and stabilization to mitigate artifacts caused by interpolation between HRTF filters, the use of time-varying HRTF filters, and spectral coloration due to crosstalk.

Description

[Technical Field]

[0001] Cross-reference of related applications This application is a continuation of U.S. Patent Application No. 18 / 424,295, now USP12,395,806, filed on 26 January 2024. U.S. Patent Application No. 18 / 424,295, now USP12,395,806, is a continuation of U.S. Patent Application No. 17 / 513,249, now USP11,924,623, filed on 28 October 2021. This application relates to U.S. Patent Application No. 17 / 513,175, now USP11,665,498, filed on 28 October 2021. These applications are incorporated herein by reference in their entirety for all purposes.

[0002] field The technology described herein relates to 3D audio, and more specifically, to signal processing techniques for improving the quality and accuracy of virtual 3D object placement in virtual sound generation systems for augmented reality, video games, and other applications. [Background technology]

[0003] background Even with only two ears, we humans can detect the 3D location of sound sources with astonishing accuracy. Sitting by the back door on a summer night, you can hear crickets chirping from the left, frogs croaking from the right, children playing behind you, and distant thunder from beyond the horizon. In a concert hall, with your eyes closed, you can hear the violins on the left, the cellos and double basses on the right, the contrabass behind the cellos, the wind instruments and violas in the middle, the woodwinds in front, the brass in the back, and the percussion behind them.

[0004] Some people believe that the reason we have evolved such sound localization capabilities is that it was critical to our survival - that is, by perceiving that a saber-toothed tiger was rustling in the grass a short distance to our right and moving toward us, we could protect ourselves from being attacked. Regardless of how and why we developed this remarkable ability to perceive sound localization, it is part of the way we perceive the world. Accordingly, when simulating reality using virtual simulations such as video games (including first-person or other immersive games), augmented reality, virtual reality, augmented reality, or other representations related to virtual soundscapes and / or 3D spatial sound, it has become desirable to model and simulate sound sources. As a result, we perceive them as having realistic spatial locations in three-dimensional space.

[0005] Lateral localization Intuitively, sounds generally heard by the left ear come from the left, and sounds generally heard by the right ear come from the right. Simple stereo panning control uses variable volume levels at the left and right headphone speakers to create the illusion that a sound is directed to the left, directed to the right, or located at the center.

[0006] The psychoacoustic mechanisms we use to detect lateral or azimuthal localization are in fact far more complex than simple stereo intensity panning. Our brain can distinguish subtle differences in both the amplitude and timing (phase) of sounds detected by the ears. The relative delay between the time a sound reaches the left ear and the time the same sound reaches the right ear is called the interaural time difference or ITD. The difference in amplitude or level between the sound detected by the left ear and the same sound detected by the right ear is called the interaural level difference or ILD. Our brain uses both ILD and ITD for sound localization.

[0007] It can be seen that one or the other (ILD or ITD) is more useful depending on the characteristics of a specific sound. For example, low-frequency (low-pitch) sounds have wavelengths larger than the dimensions of our head, so our brain can use phase (timing difference) information to detect the lateral direction of low-frequency or lower-pitch sounds. On the other hand, since the wavelengths of higher-frequency (higher-pitch) sounds are shorter, phase information is not useful for sound localization. However, our head more easily attenuates higher-frequency sounds, so our brain uses this additional information to determine the lateral location of high-frequency sound sources. In particular, our head "shadows" high-frequency sounds originating from the left side of the head from the right ear, and "shadows" high-frequency sounds originating from the right side of our head from the left ear. Our brain can detect minute differences in amplitude / level between the left and right ears based on such shadowing to localize high-frequency sounds. For mid-frequency sounds, there is a transition region that helps our brain localize sounds using both phase (timing) and amplitude / level differences.

[0008] Elevation and Front-Back Localization It is more difficult to distinguish whether a sound is coming from behind or in front of oneself. Consider a sound source directly in front of you and the same sound directly behind you. The sound emitted by the sound source reaches the left and right ears at exactly the same time in both cases. Is the sound coming from in front of you or behind you? To resolve this ambiguity, our brain relies on how our own ears, head, and body modify the spectrum of sound. Sounds originating from different directions interact with the contours of our body in different ways. The shape and size of our head, neck, shoulders, and torso, and in particular the sound reflections caused by the outer ear (or pinna), act as filters that modify the frequency spectrum of the sound reaching our middle ear.

[0009] Our brains use these spectral modifications to infer the direction of a sound source. For example, sounds approaching from the front generate resonance created by the complex folds inside the auricle, while sounds from behind are blocked by the auricle. Similarly, sounds from above can be reflected by the shoulders, while sounds from below are blocked by the torso and shoulders. The combined effects of these reflections and blockages allow our brains to apply what is essentially a direction-selective filter.

[0010] Audio spatialization system Because the way the head modifies sound is crucial to how the brain perceives the direction of sound, modern 3D audio systems attempt to model these psychoacoustic mechanisms using head-related transfer functions (HRTFs). HRTFs capture the timing, level, and spectral differences that our brains use to localize sound, and they form the basis of most modern 3D sound spatialization techniques.

[0011] HRTF is the Fourier transform of the corresponding head-impulse response (HRIR). A mono object sound x(t) is converted to each ear h L (t) and h R By convolving with HRIR at each (t) interval, binaural stereo channel y L (t) and y R (t) is created (see Figure 5). This process is performed for each of the M sound objects (Figure 5 shows three different sound objects, but any number of M can exist). Each sound object represents or models a different sound source in a 3D virtual space. Similarly, each HRTF H is created for the mono object sound X(f). L (f) and H R By multiplying by (f), that is,

number

[0012] This allows convolution to be performed in the frequency domain.

[0013] Binaural audio, a common type of 3D audio effects technique that typically uses headphones worn by the listener, uses HRTF (Head-Related Tracking Function) to send sound from the sound source to both of the listener's ears, allowing the listener to perceive the direction from which the sound is likely coming and its distance from the source. By applying different HRTFs to the left and right ears in the signal or digital domain, the brain can be tricked into believing that the sound is coming from a real sound source located in an actual 3D position in a real 3D space.

[0014] For example, in such a system, the sound pressure level (gain) of the sound heard by the listener changes with frequency until the sound reaches the listener's middle ear. In 3D audio systems, these frequency characteristics are typically processed electronically using an HRTF (Heat-Related Transformer) that takes into account not only the direct sound arriving directly at the listener's middle ear, but also the effects of sound diffracted and reflected by the auricle or pinna, other parts of the head, and other parts of the listener's body—just as real sound propagating through the air is processed.

[0015] Frequency characteristics also vary depending on the location of the sound source (e.g., direction). Furthermore, the frequency characteristics of sounds that should be detected by the left and right ears may differ. In spatial sound systems, by carefully controlling the frequency characteristics of sounds reaching the listener's left and right middle ears, their volume, and the time difference between them, it is possible to control the location (e.g., direction) of the sound source that the listener should perceive. This allows sound designers to precisely position sound sources within the soundscape, creating the illusion of realistic 3D sound. For example, see US10796540B2; Sodnik et al., "Spatial sound localization in an augmented reality environment," OZCHI '06: Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (November 2006) Pages 111-118 https: / / doi.org / 10.1145 / 1228175.1228197; and Immersive Sound: The Art and Science of Binaural and Multi-Channel Audio (Routledge 2017). [Overview of the project] [Problems that the invention aims to solve]

[0016] While much research has been conducted in the past, further improvements are possible and desirable.

[0017] A patent or application file must include at least one drawing produced in color. A copy of this patent or patent application publication containing the color drawing will be provided by the Office upon request and payment of the necessary fees. [Brief explanation of the drawing]

[0018] [Figure 1] This is a schematic block diagram of an exemplary sound capture system. [Figure 1A] Figure 1 is a flowchart illustrating exemplary program control steps performed by the system shown. [Figure 2] This is a block diagram of an exemplary sound and graphics generation system. [Figure 3] Figure 2 is a block diagram of an exemplary sound generation system. [Figure 4] Figure 2 is a flowchart illustrating exemplary program control steps performed by the system shown. [Figure 5] Exemplary spatialization parameters are shown. [Figure 6] This is a block diagram of an exemplary object-based spatializer architecture that can be incorporated into the systems shown in Figures 2 and 3. [Figure 7] An exemplary spatial interpolation region is shown. [Figure 8] This shows the desired time alignment between HRTF filters. [Figure 9] An exemplary block diagram of an exemplary delayed-compensated bilinear interpolation technique is shown. [Figure 10] This is a block diagram of an exemplary modified architecture that uses crossfading. [Figure 11] This shows the frame time window. [Figure 12] This shows a frame time window with a crossfade. [Figure 13A] A comparison of the frequency domain is shown, along with a frequency domain spectrogram without delay compensation. [Figure 13B] A comparison of the frequency domain is shown, along with a frequency domain spectrogram with delay compensation. [Figure 14A] A time-domain comparison is shown, and a time-domain plot without delay compensation is also presented. [Figure 14B] A time-domain comparison is shown, followed by a time-domain plot with delay compensation. [Figure 15] An example crosstalk pathway is shown. [Figure 16] This illustrates an exemplary crosstalk pathway in the context of spatializers. [Figure 16A] This illustrates an exemplary crosstalk pathway in the context of spatializers. [Figure 17] This shows an exemplary crosstalk path from the internal left and right loudspeakers of a handheld stereophonic (multi-channel) video game device to each of the listener's ears. [Modes for carrying out the invention]

[0019] Detailed description of non-limiting embodiments We have developed a novel object-based spatializer algorithm and associated sound processing system to demonstrate a new spatial audio solution for virtual reality, video games, and other 3D audio spatialization applications. The spatializer algorithm processes audio objects to give a compelling impression of virtual sound objects emanating from any location in 3D space when heard through headphones or other means.

[0020] The object-based spatializer applies a head-related transfer function (HRTF) to each audio object and then synthesizes all filtered signals into a binaural stereo signal suitable for headphones or other playback. A high-quality HRTF database and novel signal processing techniques enable the creation of an engaging audio playback experience that provides a strong sense of externalization and accurate object localization.

[0021] Exemplary features The following are at least some exemplary features of object-based spatializer design.

[0022] Each audio object is spatialized independently based on its object position.

[0023] Supports multiple (M) simultaneous objects.

[0024] The object's position may change over time.

[0025] Reasonable CPU load (for example, by using efficient FFT-based convolution or other techniques).

[0026] A novel delay-compensated HRTF interpolation technique.

[0027] An efficient crossfading technique that reduces artifacts caused by time-varying HRTF filters.

[0028] Exemplary sound capture system A realistic soundscape containing any M sound objects can be provided using an object-based spatializer in a video game system, an artificial reality system (such as an augmented or virtual reality system), or another system with or without graphics or image-based components. The soundscape can be defined in a three-dimensional (xyz) coordinate system. Each of the M artificial sound objects can be defined within the soundscape. For example, in a forest soundscape, a sound object for a bird high in a tree may be defined at a certain xyz position (e.g., as a point source), a sound object for a waterfall may be defined at a different xyz position or range of positions (e.g., as a surface source), and wind blowing through the trees may be defined as a sound object at a different xyz position or range of positions (e.g., as a surface source). Each of these objects may be modeled separately. For example, a bird object can be modeled by capturing the sound of a real bird, defining the xyz virtual position of the bird object in the soundscape, and processing the captured sound through an HRTF (pre-processed or during real-time playback) based on the virtual position of the bird object and the position (and possibly orientation) of the listener's head. Similarly, the sound of a waterfall object can be captured from a real waterfall or synthesized in a studio. A waterfall object can be modeled by defining the xyz virtual position of the waterfall object in the soundscape (which can be a point source or a surface source depending on how far the waterfall object is from the listener). The captured sound is then processed through an HRTF (pre-processed or during real-time playback) based on the virtual position of the waterfall and the position (and possibly orientation) of the listener's head. Any M such sound objects can be defined in the soundscape.

[0029] At least some of the sound objects can have a changeable or dynamic position (for example, a bird can be modeled to fly from one tree to another). In video games or virtual reality, the position of a sound object can correspond to the position of a virtual (e.g., visual or hidden) object in the world of 3D graphics, so a bird can be modeled, for example, by both a graphics object and a sound object that appear to be in the same virtual location to the listener. In other applications, the presence of a graphics component is not necessary.

[0030] To model sound objects, the sound source (e.g., birdsong, waterfall spray, blowing wind, etc.) is first captured from real-world sounds or artificially synthesized sounds. In some cases, real-world sounds can be digitally modified to apply various effects, such as making voices sound higher or lower, and to remove unwanted noise. Figure 1 shows an exemplary system 100 used to capture and play back sound. In this example, sound is captured using any number of real and / or virtual microphones 102 (blocks 202, 204 in Figure 1A). The sound may be digitized by an A / D converter 104, further processed by a sound processor 106 (block 206 in Figure 1A), and then stored as a sound file 109 (blocks 208, 210 in Figure 1A). In this way, any kind of sound can be captured—birdsong, waterfall, jet plane, police siren, wind blowing through the grass, human singer, voice, crowd noise, etc. In some cases, instead of capturing naturally occurring sounds, or in addition to doing so, synthesizers can be used to create sounds such as sound effects. A collection or library of the resulting sound files 109 can be stored (block 208 in Figure 1A), and this can be used to create and present one or more sound objects in a virtual 3D soundscape. Such sound libraries are often used when creating content. Often, the library defines or uses a mono sound for each object, which are then manipulated to give spatial effects as described below.

[0031] Figure 2 shows an exemplary, non-limiting sound spatialization system including visual and audio capabilities. In the example shown, a non-temporary storage device 108 stores sound files 109 and graphics files 120. A processing system 122, including a sound processor 110, a CPU 124, and a graphics processing unit 126, processes the stored information in response to input from a user input device 130 to output binaural 3D audio via stereo headphones 116 and 3D graphics via a display 128. The display 128 can be any type of display, such as a television, computer monitor, handheld display (e.g., provided on a portable device such as a tablet, mobile phone, or portable game system), goggles, or glasses. Similarly, headphones offer the advantage of providing complete control over separate sound channels reaching each of the listener's left and right ears, although in other applications, sound can be reproduced via loudspeakers (e.g., stereo, surround sound, etc.), or in some embodiments, via other transducers. Such a system can be used for real-time interactive playback of sound, or for recording sound for later playback (e.g., via podcasting or broadcasting), or both. In such cases, the virtual and relative positions of the sound object and the listener may be fixed or variable. For example, in a video game or virtual reality scenario, the listener may change its own position in the soundscape and may also be able to control the position of a sound object in the soundscape (in some embodiments, the listener's position corresponds to a viewpoint used in 3D graphics generation that provides a first-person or third-person “virtual camera” position, see, for example, USP5754660). On the other hand, the processing system may autonomously move or control the positions of other sound objects in the soundscape (“bot” control).In a multiplayer scenario, one listener may be able to control the position of several sound objects, and another listener may be able to control the position of other sound objects. In such a moving scenario, the position of the sound objects changes continuously with respect to the position of the listener's left and right ears. However, exemplary embodiments include, but are not limited to, moving objects. For example, a sound-generating object can change its position, distance, and / or orientation relative to the listener's position without the “movement” being perceived or controlled to “move” (for example, using a normal sound-generating object to give multiple instances such as numerous songbirds in a tree or numerous thunderclaps from different parts of the sky).

[0032] Figure 3 shows an exemplary, non-limiting, more detailed block diagram of a 3D spatial sound reproduction system. In the example shown, the sound processor 110 generates left and right outputs and outputs them to the respective digital-to-analog converters 112(L) and 112(R). The resulting two analog channels are amplified by analog amplifiers 114(L) and 114(R) and supplied to the left and right speakers 118(L) and 118(R) of the headphones 116, respectively. The left and right speakers 118(L) and 118(R) of the headphones 116 vibrate to generate sound waves, which propagate through the air and by conduction. These sound waves have timing, amplitude, and frequency controlled by the sound processor 110. The sound waves strike the left and right middle ears or eardrums of the listener. The middle ear vibrates in response to the generated sound waves, and the vibrations of the middle ear correspond to the frequency, timing, and amplitude specified by the sound processor 110. The human brain and nervous system detect vibrations in the middle ear and, using the brain's neural network, perceive the direction and distance, and thus the apparent spatial relationship between the virtual sound object and the listener's head, based on the frequency, amplitude, and timing of the vibrations identified by the sound processor 110, enabling the listener to perceive the sound.

[0033] Figure 4 shows an exemplary, non-limiting system flowchart of the operations performed by the processing system 122 under the control of instructions stored in the memory unit 108. In the example shown, the processing system 122 receives user input (blocks 302, 304), processes graphics data (block 306), processes sound data (block 308), and generates output to headphones 116 and display 128 (blocks 310, 312). In one embodiment, this program control flow is performed periodically, such as once per video frame (e.g., every 1 / 60th of a second or every 1 / 30th of a second). On the other hand, the sound processor 110 may process sound data multiple times per video frame processed by the graphics processor 126 (block 308). In one embodiment, an application programming interface (API) is provided which allows the CPU 124 to (a) re)write relative distance, position, and / or direction parameters (for example, a set of parameters for each sound generation object) to memory accessible by the digital signal, audio, or sound processor 110 that executes the sound data (block 308), and (b) instruct the digital signal, audio, or sound processor 110 to perform sound processing on the next block or "frame" of audio data associated with the sound generated by the sound generation object that the CPU 124 places in and / or references in main memory or other shared memory accessible by both the CPU 124 and the sound processor 110.The digital signal, audio, or sound processor 110 may thus perform a number of sound processing operations for each video frame for each of a number of localized sound-generating objects to generate multiple audio output streams, which are then mixed together or combined with other unprocessed or differently processed audio streams (e.g., music playback, character voice playback, explosions, wind sounds, and other unlocalized sound effects) to output a composite sound output to headphones that includes both localized 3D sound components and unlocalized (e.g., conventional monophonic or stereophonic) sound components.

[0034] HRTF-based spatialization In one example, the sound processor 110 uses a pair of HRTF filters to capture frequency responses that characterize how the left and right ears receive sound from their positions in 3D space. The processing system 122 can apply different HRTF filters to the left and right sound channels for each sound object and apply them to the left and right channels of the headphones 116. The responses capture important perceptual cues such as interaural time difference (ITD), interaural intensity difference (ILD), and spectral deviation, which help the human auditory system localize sound, as discussed above.

[0035] In many embodiments using multiple sound objects and / or moving sound objects, the filters used to filter the sound objects differ depending on the location of the sound objects. For example, the filter applied to a first sound object at (x1, y1, z1) is different from the filter applied to a second sound object at (x2, y2, z2). Similarly, if a sound object moves from position (x1, y1, z1) to position (x2, y2, z2), the filter applied at the start of the movement is different from the filter applied at the end of the movement. Furthermore, if sound is emitted from the object while it is moving between these two positions, the HRTF for the sound object at such an intermediate position must be properly modeled by applying corresponding different filters. Thus, for moving sound objects, the HRTF filtering information can change over time. Similarly, the virtual location of a listener in a 3D soundscape may change relative to the sound object, or (for example, in a simulation game where the listener is moving through a forest and animals or enemies are following the listener or otherwise changing position in response to the listener's position or for other reasons) the positions of both the listener and the sound object may be moving. Often, a set of HRTFs are placed at predetermined locations for the listener, and interpolation is used to model sound objects located between these predetermined locations. However, as will be explained below, such interpolation can produce artifacts that reduce realism.

[0036] Exemplary Architecture Figure 6 is a high-level block diagram of the object-based spatializer architecture. To keep processing costs as low as possible, most of the processing is done in the frequency domain, including efficient FFT-based convolutions.

[0037] Processing per object The first stage of the architecture includes a processing loop 502 for each available audio object. Thus, there may be M processing loops 502(1), ..., 502(M) for M processing objects (for example, one processing loop for each sound object). Each processing loop 502 processes sound information (e.g., an audio signal x(t)) about the corresponding object based on the sound object's position (e.g., in xyz 3-dimensional space). Both of these inputs may change over time. Each processing loop 502 processes its associated sound object independently of any processing performed by other processing loops on that sound object. The architecture is extensible, for example, by adding an additional processing loop block 502 for each additional sound object. In one embodiment, the processing loops 502 are implemented by a DSP that performs software instructions, while other implementations can use hardware or a combination of hardware and software.

[0038] The per-object processing stage involves applying the distance model 504, converting to the frequency domain using FFT 506, and applying a pair of digital HRTF FIR filters based on the unique position of each object (since FFT 506 converts the signal to the frequency domain, the application of the digital filters is a simple multiplication shown by the circle 509 labeled "X" in Figure 6) (multiplication in the frequency domain is equivalent to performing convolution in the time domain, and multiplication using typical hardware is often more efficient than performing convolution).

[0039] In one embodiment, all processed objects are located in the internal synthesis bus Y L (f) and Y R(f) are summed into 510(L) and 510(R). These synthesis buses 510(L), 510(R) accumulate all of the filtered signals for the left ear and the right ear respectively. In Figure 6, the summation of all filtered objects into the binaural stereo channels is performed in the frequency domain. Internal synthesis bus Y L (f) and Y R (f) 510 accumulate all of the filtered objects. [Equation]

[0040] where M is the number of audio objects.

[0041] Inverse FFT and overlap-add These summed signals are converted back to the time domain by inverse FFT blocks 512(L), 512(R), and the overlap-add processes 514(L), 514(R) provide an efficient way to implement convolution of very long signals (see, for example, Oppenheim et al., Digital Signal Processing (Prentice-Hall 1975), ISBN 0-13-214635-5; and Hayes, et al. Digital Signal Processing, Schaum's Outline Series (McGraw Hill 1999), ISBN 0-07-027389-8). Next, the output signal y L (t), y R (t) (see Figure 5) may be converted to analog, amplified, and applied to audio transducers at the listener's ears. As shown in Figure 6, the inverse FFT 512 is applied to each of the internal synthesis buses Y L (f) and Y R (f). The forward FFT for each object is zero-padded up to double its length, resulting in an FFT length of N. A typical overlap-add technique using a 50% overlapping window as shown in Figure 11 can achieve valid convolution, such that the final output channel yL (t) and y R (t)

[0042] Distance Model 504 Each object is attenuated using a distance model 504 that calculates attenuation based on the relative distance between the audio object and the listener. Thus, the distance model 504 attenuates the audio signal x(t) of a sound object based on how far the sound object is from the listener. Distance model attenuation is applied in the time domain and includes a frame-to-frame slope to avoid discontinuities. The distance model can be configured to use a linear and / or logarithmic attenuation curve or any other suitable distance attenuation function. Generally, the distance model 504 applies higher attenuation to a sound x(t) as it travels a greater distance from the object to the listener. For example, the attenuation rate may be affected by the medium through which the sound travels (e.g., air, water, a dense forest, a rainy scene, etc.).

[0043] FFT506 In one embodiment, each attenuated audio object is converted to the frequency domain via FFT506. This conversion to the frequency domain results in a more optimized filtering implementation in most embodiments. Each FFT506 is zero-padded to twice its original value to prevent cyclic convolution and to accommodate an FFT-based superposition implementation.

[0044] HRTF interpolation 508 For a compelling and immersive experience, it is useful to achieve smooth, high-quality sound from any position in 3D space. Digital HRTF filters are typically defined for predefined directions captured in an HRTF database. Therefore, such a database may have a lookup table of HRTF parameters for each of numerous xyz locations in the soundscape coordinate system (recall that in one embodiment using a distance function, distance is considered). If the desired direction for a given object does not perfectly match the predefined directions in the HRTF database (i.e., the vector between the sound object location and the listener location in the soundscape coordinate system), interpolation between HRTF filters can enhance realism.

[0045] HRTF bilinear interpolation HRTF interpolation is performed twice, using different calculations for the left and right ears. Figure 7 shows an example of a region in soundscape space (represented here in polar or spherical coordinates), where filters are defined at the four corners of the area (region), and the location of the sound object and / or the direction of the sound are defined within that area / region. In Figure 7, azimuth represents the horizontal dimension on the sphere, and elevation represents the vertical dimension on the sphere. One possibility is to simply take the nearest neighbor—that is, to use a filter defined at the corner of the area closest to the location of the sound object. This is very efficient as it does not require any calculations. However, the problem with this method is that it creates a filter function that is discontinuous to a perceptible degree. If the sound object is moving within the soundscape, the sound characteristics sound as if they are "jumping" from one set of filter parameters to another, creating a perceptible artifact.

[0046] A better technique for interpolating HRTFs on a sphere is to use a non-zero-order interpolation method. For example, bilinear interpolation interpolates between four filters defined at the angles of the domain, based separately on the distance for each dimension (azimuth and elevation).

[0047] The desired orientation relative to the object is defined in spherical coordinates by the azimuth angle θ and elevation angle φ. The desired orientation corresponds to the HRTF filter H θ1,φ1 (f), H θ1,φ2 (f), H θ2,φ1 Assume that this refers to the interpolation region defined by four endpoints (θ1,φ1), (θ1,φ2), (θ2,φ1), and (θ2,φ2) where (f) and Hθ2,φ2(f) are provided. Assume that θ1 < θ2 and φ1 < φ2 and θ1 ≤ θ ≤ θ2 and φ1 ≤ φ ≤ φ2. Figure 7 illustrates this scenario.

[0048] Interpolation involves determining coefficients for each of the two dimensions (azimuth and elevation) and using these coefficients as weights for the interpolation calculation. α θ and α φ These are linear interpolation coefficients calculated separately for each dimension, as follows.

number

[0049] The resulting bilinear interpolation HRTF filter is as follows:

number

[0050] Nevertheless, the bilinear interpolation technique described above can have problems. ITD is one of the important perceptual cues captured and reproduced by the HRTF filter, and therefore, time delays are usually observed between filters. Adding time-delayed signals can be problematic, resulting in artifacts and cancellations such as comb filtering. If the time delays between adjacent HRTF filters are large, the quality of interpolation between those filters deteriorates significantly. The left side of Figure 8 shows such exemplary time delays between four filters defined at each of the four corners of the bilinear domain. Because their timings are different, the values ​​of the four filters shown when synthesized through interpolation result in a "blurred" waveform with components that may interfere with each other in a frequency-dependent manner, either reinforcing or canceling each other out. This creates undesirable frequency-dependent audible artifacts that reduce the fidelity and realism of the system. For example, as the sound object position moves between the filter locations in Figure 7, a perceptible comb filtering effect may be heard, appearing as if the amplitude changes or modulates up and down for different frequencies in the signal.

[0051] Figure 14A shows such a comb filter effect in a time-domain signal waveform, and Figure 13A shows such a comb filter effect in a frequency-domain spectrogram. These figures illustrate audible modulation artifacts when a sound object moves from a position perfectly aligned to the filter locations to a position between multiple filter locations (e.g., equidistant). Note the striping effect in the spectrogram of Figure 13A and the corresponding peaks in the time-domain signal of Figure 14A. Thus, significant artifacts can be heard or seen with standard bilinear interpolation, which is highlighted, in one example, by the relatively low angular resolution of the HRTF database at an elevation angle of 15 degrees.

[0052] A better approach: Delay-compensated bilinear interpolation To address the interpolation problem between time-delayed HRTF filters, a new technique called delay-compensated bilinear interpolation has been developed. The idea behind delay-compensated bilinear interpolation is to time-align the HRTF filters before interpolation so that aggregate artifacts are largely avoided, and then time-shift the interpolated result back to the desired time position. In other words, even if HRTF filtering is designed to give the exact amount of time delay to create different spatial effects from one filter position to another, one exemplary implementation would make the time delays "all the same" for the four filters being interpolated, perform the interpolation, and then, after the interpolation is done, further time-shift the result to restore the timing information that was removed for the interpolation.

[0053] An example of a desired time alignment between HRTF filters is shown in Figure 8. Specifically, the left side of Figure 8 shows the original HRTF filter as it would be stored in an HRTF database, and the right side of Figure 8 shows the same filter after selective time shifting has been applied to delay-compensate for the HRTF filter in the interpolation region.

[0054] By multiplying the HRTF filter by an appropriate complex exponent, time shifting can be efficiently achieved in the frequency domain. For example,

number

[0055] Figure 9 is a block diagram of an exemplary delayed-compensated bilinear interpolation technique. This technique applies an appropriate time shift 404 to each of the four HRTF filters, then applies a standard bilinear interpolation 402, and then applies a post-interpolated time shift 406. Note that the pre-interpolated time shift 404 is independent of the desired direction (θ,φ) in the interpolation region, while the bilinear interpolation 402 and post-interpolated time shift 406 depend on (θ,φ). In some embodiments, it may not be necessary to time shift all four filters—for example, one of the filters may remain time-static, while three (or any other number) of the other filters may be time-shifted relative to the time-static filter. In other embodiments, all four (or any other number) HRTF filters may be time-shifted as shown in Figure 9.

[0056] A delayed-compensated bilinear interpolation filter can be calculated as follows (the bilinear interpolation calculation is the same as in the previous example, except that multiplication by a complex exponential sequence is added to every filter):

number

[0057] The complex exponential term mathematically defines the time shift, and different time shifts are applied to each of the four weighted filter terms. One embodiment computes such a complex exponential column in real time. Another embodiment stores the pre-computed complex exponential column in an indexed lookup table and accesses (reads) the pre-computed complex exponential column or values ​​representing or derived therefrom from the table.

[0058] Efficient time shifting for delayed-compensated bilinear interpolation To perform time shifting for delayed-compensated bilinear interpolation, the HRTF filter should contain a complex exponential sequence.

number

[0059] The solution used in the current implementation is to utilize the cyclic relationship between cosine and sine functions. x[n]=2cos(a)x[n-1]-x[n-2] This can be written as follows: In the formula, a represents the frequency of the sequence. Therefore, our desired complex exponential sequence

number

number

number

number

[0060] Time shift decision Delay-compensated bilinear interpolation 402 applies a time shift to the HRTF filter to achieve time alignment before interpolation. This then raises the question of which time shift value should be used to give the desired alignment. In one embodiment, offline or online analysis is used to determine a suitable time shift m for each interpolation region. θi,φjThe time shift can be calculated in advance. In other embodiments, the time shift can be determined dynamically in real time. The analysis performed for one exemplary current implementation uses so-called fractional cross-correlation analysis. This fractional cross-correlation technique is similar to standard cross-correlation but includes a fractional sample delay. Using the fractional delay along with the maximum cross-correlation, a time shift that can give a suitable time alignment is derived. The time shift m is calculated in advance for each interpolation region. θi,φj The implementation may include a lookup table which can be used during runtime for each interpolation calculation. Such a table can be stored in firmware or other non-volatile memory and accessed on demand. Other implementations can use combinational logic or other logic to generate appropriate values.

[0061] all m θi,φj By using appropriately selected values ​​for (see below), the time delay between HRTF filters can be compensated, and all HRTF filters can be effectively time-aligned before interpolation. See the right side of Figure 8. However, if the desired direction is the interpolation endpoint (θ) i ,φ j If the original time shift amount m is perfectly aligned, it is desirable that the time delay resulting from the interpolation filter transitions smoothly across the interpolation region and approaches the unchanged filter response. Thus, the original time shift amount m θ1,φ1 , m θ1,φ2 , m θ2,φ1 , m θ2,φ2 Based on this, the interpolated filter can be time-shifted again by the interpolation amount.

number

number

[0062] This interpolated time shift 406 is in the opposite direction to the original time shift 404 applied to the HRTF filter. This allows for an unaltered response to be achieved when the desired direction is perfectly spatially aligned with the interpolation endpoint. In this way, the additional time shift 406 restores the timing to an unaltered state, preventing timing discontinuities when moving away from near-perfect alignment with a particular filter.

[0063] The overall result of the delay-compensated bilinear interpolation technique is that it can effectively time-align filters during interpolation to help avoid aggregate artifacts, while also enabling smooth transitions of time delays across the interpolation region and achieving a unchanged response at the outermost interpolation endpoints.

[0064] Effectiveness of delayed-compensated bilinear interpolation An object rotating around the listener's head in the frontal plane is observed as a good demonstration of the effectiveness of the delayed-compensated bilinear interpolation technique. Figures 13A, 13B, 14A, and 14B show exemplary results of a white noise object rotating around the listener's head in the frontal plane using both standard bilinear interpolation and delayed-compensated bilinear interpolation. Figures 13B and 14B show exemplary results using delayed-compensated bilinear interpolation, where the signal is significantly smoother or less "striped," reducing or eliminating the aforementioned comb filter effect. Thus, artifacts are virtually avoided when using delayed-compensated bilinear interpolation.

[0065] Architecture using crossfades Thus, the types of time-varying HRTF FIR filters discussed above are parameterized using parameters that represent the relative position and / or distance and / or direction between the sound-generating object and the listener. In other words, when the parameters representing the relative position and / or distance and / or direction between the sound-generating object and the listener change (for example, due to changes in the position of the sound-generating object, the listener, or both), the filter properties of the time-varying HRTF filter change. It is known that such changes in filter properties can lead to processing artifacts if not handled properly. For example, see Keyrouz et al., "A New HRTF Interpolation Approach for Fast Synthesis of Dynamic Environmental Interaction," JAES Volume 56 Issue 1 / 2 pp.28-35; January 2008, Permalink: http: / / www.aes.org / e-lib / browse.cfm?elib=14373; and Keyrouz et al., "A Rational HRTF Interpolation Approach for Fast Synthesis of Moving Sound," 2006 IEEE 12th Digital Signal Processing Workshop & 4th IEEE Signal Processing Education Workshop, 24-27 Sept. 2006 DOI:10.1109 / DSPWS.2006.265411.

[0066] To mitigate artifacts from time-shifted FIR filters, exemplary embodiments provide a modified architecture that utilizes crossfading between filter results, as shown in the block diagram of Figure 10. In one embodiment, all processing blocks are the same as those described in the previous section, but the architecture is modified to generate two sets of binaural stereo channels for each frame. However, in other embodiments, two binaural stereo signals can be generated in any desired manner (for example, not necessarily using the time-shifted bilinear interpolation architecture of Figure 9), and crossfading as described below can be applied to provide a smooth transition from one HRTF filter to the next. In other words, the crossfader 516 in Figure 10 solves a different discontinuity problem than that solved by the configuration of Figure 9, namely, the discontinuity that results from convolving the results of two very different HRTF filter transformations based on a sound object (or listener or both) that rapidly changes position from one frame to the next. This is a problem independent of the issues addressed using the time-shifting techniques described above, and one technique does not necessarily depend on the other; each technique can be used in its own implementation without the other. Nevertheless, both techniques can be advantageously used in common implementations to avoid both types of discontinuities and associated perceptible artifacts.

[0067] Frame delay The crossfade architecture in Figure 10 includes a frame delay for the HRTF filters. As a result, there are four HRTF filters per object: i.e., selected based on the current object position. L (f) and H R (f) and a delay filter from the previous frame based on the previous object position.

number

number

[0068] All four HRTF filters are used to filter the current sound signal occurring in the current frame (i.e., in one embodiment, this is not a case where the filtering results of the previous frame can be stored and reused—rather, in such an embodiment, the current sound signal for the current frame is filtered using two left-hand HRTF filters and two right-hand HRTF filters, where one pair of left / right HRTF filters is selected or determined based on the current position of the sound object and / or the current orientation between the sound object and the listener, and the other pair of left / right HRTF filters are the same filters used in the previous frame time). Another perspective: at a given frame time, the HRTF filters or parameterized filter settings selected for that frame time are reused in the next or subsequent frame time to mitigate artifacts resulting from changing the HRTF filters from a given frame time to the next or subsequent frame time. In the example shown, such a configuration is extended to include HRTF filter interpolation, HRTF filtering operations, multi-object signal summation / combination, and inverse FFT from the frequency domain to the time domain across all sound objects.

[0069] As a result of adding a frame delay filter, the same HRTF filter is applied to two consecutive frames, in which case it is guaranteed that there are no artifacts in the superimposed summation region for their outputs. This architecture outputs a suitable superimposed frame (see Figure 11) that can together crossfade to produce a smooth transition. In this context, the term “frame” may include or mean a portion of an audio signal stream that contains at least one audio sample, such as a portion consisting of N audio samples. For example, there may be multiple audio “frames” associated with a video frame with a duration of 1 / 60 or 1 / 30 of a second, and each audio frame contains a number of audio samples to be processed. As described above, in the exemplary embodiment, the system does not store and reuse the previous filtered output or result, but instead applies a parameterized filtering operation of the previous filtering operation (based, for example, on the relative positions between the previous and now changed sound-generating objects and the listener) to newly incoming or current audio data. However, in other embodiments, the system may use both the result of the previous filtering operation and the parameters of the previous filtering operation to develop the current or new audio processing output. Thus, the applicant does not intend to abandon the use of previously generated filter results for various purposes, such as those known to those skilled in the art.

[0070] Crossfade 516 Each crossfader 516 (which operates in the time domain after the associated inverse FFT block) controls the two filtered signals.

number

number

number

number

[0071] In one embodiment, the crossfader 516 comprises a dual gain controller (e.g., a multiplier) and a processor-controlled mixer (adder), where the dual gain controller increases the gain of one input by a certain amount and simultaneously decreases the gain of the other input by the same amount. In one exemplary embodiment, the crossfader 516 operates on a single stereo channel (e.g., one crossfader for the left channel and another for the right channel) and combines two variable inputs into that channel. The gain functions of each input do not need to be linear—for example, the amount by which the crossfader increases the gain of one input does not need to match the amount by which the crossfader decreases the gain of the other input. In one embodiment, the gain functions of two gain elements G1 and G2 may be G1=0 and G2=x in one setting used at the beginning of the frame (or the first half of the frame), and G1=y and G2=0 in a second setting used at the end of the frame (or the second half of the frame), and by providing an intermediate mixing value between these two time instants, some amount of the G1 signal and some amount of the G2 signal are combined together in the frame.

[0072] In one embodiment, the output of each crossfader 516 is thus entirely the result of frame-delay filtering at the beginning of the frame (or the first or first half of the frame), and therefore entirely the result of current (non-frame-delay) filtering at the end of the frame (or the second or second half of the frame). In this way, since one interpolation block produces the result of the filtering value of the previous frame while another interpolation block produces the result of the filtering value of the current frame, there is no discontinuity at the beginning or end of frame time between these two endpoints, and the crossfader 516 produces a composite of these two values. This composite begins, and mostly, as the result of frame-delay filtering, and ends, and mostly, and then entirely, as the result of non-frame-delay (current) filtering. This is illustrated in Figure 12 by the “red” (thick solid line), “blue” (dashed line), and “green” (thin solid line) trajectories of the signal.

number

[0073] Windows w(n) and w of length N D (n) (using the discrete-time exponent n) is defined as follows:

number

[0074] In one embodiment, such crossfading operation as described above is performed for each audio frame. In another embodiment, such crossfading operation is performed selectively only when audio artifacts are likely to occur, such as when a sound object changes position relative to the listening position and thus changes the filtering parameters, including but not limited to cases where a sound generating object and / or listener moves between positions.

[0075] Details of exemplary implementations In one example, the sample rate of the described system may be 24kHz, 48kHz, 60kHz, 99kHz, or any other rate; the frame size may be 128 samples, 256 samples, 512 samples, 1024 samples, or any preferred size; the FFT / IFFT length may be 128, 256, 512, 1024, or any other preferred length, and zero padding may be included if the FFT / IFFT length is longer than the frame size. In one example, each sound object may call one forward FFT, using a total of four inverse FFTs for a total of M+4 FFT calls, where M is the number of sound objects. This is relatively efficient and allows a large number of sound objects to be handled using a standard type of DSP that is equipped on many common platforms.

[0076] Additional enhancement features HRTF Personalization Head size and ITD clues HRTF is known to vary considerably from person to person. ITD is one of the most important localization cues and is highly dependent on head size and shape. Ensuring accurate ITD cues can substantially improve the spatialization quality for several listeners. ITD tuning can be performed in the current architecture of object-based spatializers. In one embodiment, ITD tuning can be achieved by multiplying the frequency-domain HRTF filter by a complex exponential sequence. Optimal ITD tuning can be derived from head size estimation or an interactive GUI. Camera-based head size estimation techniques can be used. The HRTF can be modified or customized for a given listener using sampling by placing microphones on both of their ears.

[0077] Head tracking Head tracking can be used to enhance the realism of virtual sound objects. Gyroscopes, accelerometers, cameras, or several other sensors may be used. See, for example, US10,449,444. In virtual reality systems that track the listener's head position and orientation (posture) using MARG or other technologies, head tracking information can be used to improve the accuracy of HRTF filter modeling.

[0078] Crosstalk elimination While binaural stereo audio is intended for playback through headphones, crosstalk reduction is a technique that enables playback of binaural audio through stereo speakers. By combining crosstalk reduction algorithms with binaural spatialization techniques, it's possible to create a captivating experience for stereo speaker playback.

[0079] Use of head-related transfer functions In one exemplary embodiment, a 3D audio effect is simulated using a head-related transfer function (HRP) to generate sound to be output from a sound output device. It should be noted that the sound may also be generated based on a function that assumes and calculates the sound entering the left and right ears of a listener at a predetermined listening position from a sound object. Alternatively, sound may be generated using a function other than a HRP, thereby providing the listener with a sense of sound localization. For example, the 3D audio effect may be simulated using another method to achieve an effect similar to that of binaural recording, such as holophonics or otophonics. Furthermore, in the 3D audio effect technology using HRPs in the above exemplary embodiment, the sound pressure level is controlled according to frequency until the sound reaches the middle ear from the sound object, and also based on the location (e.g., orientation) where the sound object is placed. Alternatively, the sound may be generated using any type of control. In other words, the sound to be output from the sound output device may be generated using only a function to control the sound pressure level according to frequency until the sound reaches the middle ear from the sound object, or the sound to be output from the sound output device may be generated using only a function to control the sound pressure level based on the location (e.g., orientation) where the sound object is placed. Alternatively, the sound to be output from the sound output device may be generated using only a function to control the sound pressure level using, in addition to these functions, at least one of the following: the difference in volume, the difference in transmission time, the change in phase, the change in reverberation, etc., corresponding to the location (e.g., orientation) where the sound object is placed. Alternatively, as an example of using a function other than the head-related transfer function, 3D audio effects may be simulated using a function that changes the sound pressure level according to the distance from the location where the sound object is placed to the listener. Alternatively, 3D audio effects may be simulated using a function that changes the sound pressure level according to at least one of the following: atmospheric pressure, humidity, temperature, etc., of the real space where the listener is operating the information processing device.

[0080] Furthermore, if binaural recording is used, ambient sounds recorded through a microphone embedded in a dummy head representing the listener's head or a microphone attached inside a person's ear may be used to generate the sound to be output from the sound output device. In this case, the state of the sound reaching the listener's middle ear is recorded using the listener's skull and auditory organs, or a structure similar to the skull and auditory organs themselves, thereby providing the listener with a similar sense of sound localization.

[0081] Furthermore, the sound output device does not have to be headphones or earphones that output sound directly to the listener's ears; it may also be a stationary loudspeaker that outputs sound into the real space. For example, when using stationary loudspeakers, monitors, etc., as sound output devices, multiple such output devices can be placed in front of and / or around the listener, and sound can be output from each device. As a first example, if a pair of loudspeakers (so-called 2-channel loudspeakers) are placed in front of the listener and to the left and right, sound generated by a general stereo system can be output from the loudspeakers. As a second example, if five loudspeakers (so-called 5-channel loudspeakers or "surround sound") are placed in front of, behind, to the left and right of the listener (and also in the center), stereo sound generated by a surround system can be output from the loudspeakers. As a third example, if multiple loudspeakers (for example, 22.2 multi-channel loudspeakers) are placed in front of, behind, to the left and right, and above and below the listener, stereo sound using a multi-channel sound system can be output from the loudspeakers. As a fourth example, the sound generated by the binaural method described above can be output from a loudspeaker using a binaural loudspeaker. In any of these examples, the sound can be localized in front of, behind, to the left and right, and / or above and below the listener. This allows the localization position of vibrations to be shifted using the localization position of the sound. See U.S. Patent No. 10,796,540, incorporated herein by reference.

[0082] The descriptions herein rely on certain operations in the frequency domain (e.g., fractional time shifting), but the same or similar operations may be possible in the time domain. Similarly, the descriptions herein rely on certain operations in the time domain (e.g., crossfading), but the same or similar operations may be possible in the frequency domain. Likewise, the implementations herein are DSP-based in software, but some or all of the operations can be performed in hardware or in a combination of hardware and software.

[0083] Crosstalk elimination The intended binaural stereo audio playback is for each audio channel to be reproduced independently in each corresponding ear of the listener. Specifically, through headphones, earphones, etc., the left channel is delivered only to the listener's left ear, and the right channel is delivered only to the right ear.

[0084] Binaural stereo signals are typically generated through binaural recording or spatialization processing based on HRTF, and localization cues are originally captured as ILD, ITD, and spectral filter cues. In a proper binaural reproduction, the listener can experience a compelling virtualization of the real-world sound field or soundscape, where accurate sound pressure levels are recreated for each of the listener's ears. Headphones offer a high-quality sound listening experience because they can selectively deliver sound to one or the other ear of the listener and also isolate the two ears from each other.

[0085] While headphones are commonly and successfully used for playing binaural stereo audio, there are situations where playback via loudspeakers is desirable. For example, see the diagram showing a handheld video game console with a left and right loudspeaker. Instead of using headphones or earphones, the user can listen to the sound produced by the handheld video game console using these loudspeakers. However, attempting to play binaural audio with stereo loudspeakers presents significant problems. As shown in the diagram, the sound from the left loudspeaker reaches not only the left ear but both of the listener's ears. Similarly, for the right loudspeaker, the sound from the right loudspeaker reaches not only the right ear but both of the listener's ears.

[0086] The sound transmitted from the left loudspeaker to the right ear and the sound transmitted from the right loudspeaker to the left ear are known as "crosstalk." In the context of headphone playback, it is certain that sound directed to the left ear will only go to the left ear, and sound directed to the right ear will only go to the right ear. In contrast, in the context of playback in free space, some of the sound directed to the left ear now goes to the right ear, and some of the sound directed to the right ear now goes to the left ear. Such unintended crosstalk can significantly degrade the quality of the binaural listening experience.

[0087] Crosstalk rejection is a well-known technique that attempts to mitigate the problem of crosstalk in binaural reproduction with loudspeakers by acoustically removing unwanted crosstalk in each of the listener's ears. For example, phase-shifted sound "leaking" from the left channel to the right ear can be supplied to remove leaked or misdirected sound. Similarly, phase-shifted sound "leaking" from the right channel to the left ear can be supplied to remove leaked or misdirected sound. Such rejection techniques are highly effective in reducing crosstalk.

[0088] Figures 15, 16, and 17 illustrate binaural reproduction scenarios using stereo loudspeakers.

[0089] y L (t) and y R (t) is the left and right channel of the binaural signal, h LL (t), h RR (t), h LR (t), and h RL Let (t) be the impulse response of the corresponding ipsilateral and contralateral paths from the stereo loudspeaker to each of the listener's ears. The signal z reaching the listener's ears L (t) and z R (t) is a combination of ipsilateral and contralateral paths, and can be written as follows:

number

number

[0090] If the goal is to accurately reproduce the binaural channels in each listener's ear, then the presence of crosstalk pathways hinders this goal. To improve binaural reproduction, the resulting signal reaching the listener's ear is equal to the original binaural channel itself, after the crosstalk pathway has been modified.

number

number

[0091]

number

[0092] Solving these equations for the corrected channels yields the following:

number

[0093] Therefore, if the characteristics of the stereo loudspeakers used for reproduction (i.e., the transfer functions of the ipsilateral and contralateral channels) are known in advance, reasonable binaural perception using stereo loudspeakers can be achieved with some relatively simple signal modifications.

[0094] Subjective evaluation: spectral coloration and object instability We implemented and tested our previous formulation for binaural reproduction using loudspeakers. In subjective evaluations, listeners reported that they could reasonably accurately localize the spatial position of objects; however, they reported significant coloration of the object audio and some instability in object movement.

[0095] We focused on the amount of coloration and spectral shape as factors dependent on the position of virtual objects in the soundscape. In particular, coloration was reported to be most significant for objects on the listener's side (e.g., object positions close to (±90°, 0°)). In addition, we focused on the movement of unstable objects for object positions near the median plane (e.g., positions with azimuth angles close to 0°). Here, relatively small lateral movements deviating from the median plane may be perceived as larger lateral movements than expected, resulting in an exaggeration of localization.

[0096] The perceived artifacts of coloration and instability can be explained by acknowledging that some assumptions used in the formulation of crosstalk rejection may not be entirely valid. While previous formulations for crosstalk rejection imply that a "perfect" reproduction in the listener's ear can be achieved, in reality there are numerous non-idealities that can impair the accuracy of the reproduction.

[0097] The first non-ideality that degrades or may degrade performance is the incomplete modeling of ipsilateral and contralateral transfer functions. For many use cases, fully characterizing ipsilateral and contralateral transfer functions is difficult, if not impossible, due to unknown and / or variable listener anatomical features (e.g., head shape and size), unknown and / or variable listener position and alignment, and unknown and / or variable listening environment (e.g., significant nearby reflections). Typically, it is necessary to use generalized ipsilateral and contralateral transfer functions that are reasonably accurate for the broadest possible population. The drawback of using generalized transfer functions is the reduced accuracy of crosstalk rejection for specific users in specific environments.

[0098] A second non-ideality that degrades or can degrade performance is the non-ideal nature of real-world acoustics. Acoustic signals are usually modeled as linear sound waves following superposition, but in reality, the sum of acoustic waves in real-world environments is complex and not necessarily linear. For example, this phenomenon leads to the common use of nonlinear panning laws when trying to enhance panning between a pair of loudspeakers.

[0099] To ensure users have a high-quality binaural experience with loudspeakers, solutions to mitigate color and instability artifacts are desirable. Since color and instability artifacts were perceived as being dependent on the object's position, a per-object processing technique was developed to reduce artifacts based on the unique position of each object.

[0100] Object-specific techniques Instead of reproducing a fully pre-rendered binaural signal consisting of any number of objects at arbitrary positions, consider a use case of binaural reproduction by a loudspeaker for objects processed by a spatializer algorithm. Figure 16 shows a mono audio object x i (t) is processed by the spatializer algorithm into left and right HRTF filters hi,L (t) and h i,R This shows the scenario that is convolved by (t).

[0101] For the above scenario, the signal reaching the listener's ear for a given object i can be expressed as follows:

number

[0102] The resulting signal that reaches the listener's ear is equal to the monophonic audio object convolved by the original HRFT filter, H i,L (f) and H i,R (f) is the modified HRTF filter. That is, Z L (f) = X i (f)H i,L (f) and Z R (f) = X i (f)H i,R (f) is the same as listening with headphones.

number

[0103] Solving the modified HRTF filter yields the following result.

number

[0104] Thus, in the spatializer algorithm, the modified HRTF filter is applied to each object.

number

number

[0105] Since different perceptible artifacts are observed depending on the object's position, we will examine object-specific techniques intended to mitigate such artifacts. The following two sub-sections describe additional techniques that can be used to further analyze and modify HRTF filters in an attempt to reduce prominent artifacts.

[0106] Stability per object In the previous section, object instability was identified as an artifact arising from binaural reproduction using loudspeakers. In particular, it was reported that small lateral movements of objects near the midline could correspond to an exaggerated perception of localization.

[0107] To add some perceived stability to the lateral movement of objects near the midline, the solution is to combine the crosstalk-deselected HRTF with the original HRTF in a position-dependent manner. Specifically, it is noted that for positions near the midline, the interauricular localization cues of ITD and ILD are small, and instead, reliance is placed on the auricle spectral filter cues as the primary localization cue. Since incorporating crosstalk deselection into the HRTF would likely result in excessive localization near the midline, one exemplary embodiment reduces the exaggerated crosstalk-deselected cue by partially combining the original HRTF with the crosstalk-deselected HRTF while preserving the original auricle spectral cues. The crosstalk-deselected HRTF and the original HRTF are crossfaded in a position-dependent manner as follows:

number

[0108] Here, ε is a subjectively adjustable parameter between 0 and 1 that controls the maximum amount of original HRTF to be synthesized. i The exponent in the formula is sin(θ) i )cos(φ i The ) term corresponds to the Cartesian y-coordinate of the object's position. Therefore, the HRTF with all crosstalk removed is used for the object on the listener's side, while the original HRTF is increasingly weighted as the object approaches the midline. A value close to ε=0.5 successfully improves the object's stability during subjective testing.

[0109] Equalization per object In the previous section, spectral coloration was identified as another significant artifact arising from binaural reproduction using loudspeakers. Specifically, spectral coloration is reported as position-dependent, in which case the shape and amount of coloration change based on the position of the object. Therefore, static position-independent filters are insufficient to mitigate the coloration artifact. To reduce coloration, consider calculating and applying a dynamic position-dependent equalization filter for each virtual sound-generating object in the virtual soundscape. This is possible in exemplary embodiments because the sound generation system can utilize parameters that allow it to derive the directivity and length of the path between each virtual sound generator and the user's ear. For example, in one embodiment, the sound generation system has access to the virtual position or location of each virtual sound source in 3D virtual space, and possibly other information about the sound source such as size / dimensions. From this position or location information, the sound generator can determine the length and direction of the path between each virtual sound source and each of the user's ears. Next, the FIR filter used to provide spatialization can be parameterized / customized / modified to vary the frequency response for each virtual sound source or object, thereby avoiding or reducing spectral coloration artifacts resulting from crosstalk rejection. Exemplary embodiments can model how this would sound to the user and use this to predict what the user would hear with such coloration. The goal may be to make the user hear the same frequency response that they would hear if listening with headphones (without crosstalk). In one embodiment, the sound generator internally models how the sound produced by a particular object is likely to sound to a user without headphones, and compensates the HRTF filter for that sound produced by that object for the spectral changes resulting from crosstalk compensation for that particular object. Such equalization depends on the location of each sound generation object because the HRTF applied to the sound produced by such a sound generation object is location-dependent.The equalization curves applied to different sound generation locations will be different because the HRTF filtering function applied to spatialization depends on the sound generation location.

[0110] Equalization curves are created from predictions to equalize the coloration based on the (relative) position of the sound objects. Thus, the same or similar equalization can be applied to each sound source object having the same or similar (relative) position to obtain the same or similar frequency response that the user would hear with or when headphones are used. Therefore, in some embodiments, the same equalization can be applied to all objects in a region of a 3D soundscape, and different equalizations can be applied to objects in different soundscape regions. The size and extent of such areas can be defined as needed to provide the desired accuracy. However, in the above disclosure, bilinear interpolation provides a unique HRTF for each unique sound-generating object location. Therefore, in exemplary embodiments, as the sound-generating object changes position, the HRTF changes accordingly—and consequently, the spectral coloration equalization (crosstalk removal compensation) also changes. In one example, coloration equalization is integrated as part of HRTF filtering, providing a low-overhead solution that uses the same FIR filtering operation to both provide spatial sound effects and compensate (equalize) these spatial sound effects for crosstalk effects.

[0111] Furthermore, in exemplary embodiments, equalization is independent of the specific sound effects and the (non-location) properties of the virtual sound generation objects that produce those specific sound effects (e.g., music, voices, engine sounds, etc.). Thus, in exemplary embodiments, the equalization process is applicable to any game or other sound generation presentation.

[0112] Exemplary practical implementation examples As mentioned above, due to the impracticality of the real world, there is a high probability that an error exists between the true sound pressure level at the listener's ear and the modeled sound pressure level. This error may be one of the causes of the significant coloration (e.g., changes in frequency response) that is actually observed. σ i (f) is an equalization filter calculated from the energy levels of the original HRTF and the modified HRTF as follows:

number

[0113] The denominator of the equalization filter is designed to normalize the approximate total energy of the ipsilateral and contralateral paths. When applied to a modified HRTF filter, this equalization filter results in the total energy of the modified HRTF reproduced by the listener's ear being approximately equal to the total energy of the original HRTF for each frequency band. In localization theory (e.g., Gerzon's meta-theory of localization, duplex theory), it is widely accepted that different perceptual cues are used in different ways in different frequency domains. Phase-delay cues are primarily associated with low frequencies (e.g., <800 Hz) where head dimensions are smaller than half a wavelength of the sound wave. Intensity-difference cues are primarily associated with high frequencies (e.g., >1500 Hz) where significant head shadowing effects are observed.

[0114] Since the human auditory system is known to employ different mechanisms in different frequency domains, we consider applying equalization filters in different ways in different domains. Gerzon's metatheory suggests that the velocity model is effective for frequencies below 700 Hz, where signal amplitude gain is used to derive localization, while the energy model is effective for frequencies above 1 kHz, where signal energy gain is used. Motivated by this, the frequency-dependent application of equalization filters was tested. By applying the following equation, a significant improvement in perceived coloration was observed for all object positions.

number

number

[0115] Details of exemplary implementations As shown in the exemplary handheld stereophonic (multichannel) gaming device in Figure 17, it is possible to know with high precision and certainty where the left and right stereophonic speakers of the handheld device are located. In addition, since the form factor of the handheld device is known, it is possible to predict with reasonable certainty how the user will hold the handheld gaming device, the direction and length of the path between the left and right loudspeakers and the user's left and right ears, and the emission characteristics such as the directivity of the left and right speakers. Therefore, it is possible to predict the respective locations of the left and right stereophonic speakers relative to the user's left and right ears (i.e., by predicting where the user's head is located relative to the handheld device), as well as other characteristics and factors that affect crosstalk from the left speaker to the user's right ear and from the right speaker to the user's left ear. Thus, the exemplary embodiment employs modeling that takes advantage of the physical constraints imposed by the form factor and the limited range of operating modes of a particular set of handheld video game devices. The techniques described herein can work with any device, but are even more effective when used with uniform devices such as one or a few different handheld video game devices that exhibit known uniform geometric shapes and characteristics.

[0116] Algorithm parameters

[0117] [Table 1]

[0118] All patents, patent applications, and publications cited herein are invoked by reference as if they were expressly stated, for all purposes.

[0119] While the present invention has been described in relation to what is considered to be the most practical and preferred embodiment at present, it should be understood that the invention is not limited to the disclosed embodiments, but rather intended to encompass a variety of modifications and equivalent configurations that fall within the spirit and scope of the appended claims.

Claims

1. It is a method, Associating the first position information with the first sound generation virtual object, The method comprises applying a first HRTF filtering parameterized by the first position information to filter the first multichannel sound associated with the first sound generation virtual object, A method wherein the first HRTF filtering is configured to equalize the filtered first multichannel sound based on the first position information to compensate for spectral coloration of the filtered first multichannel sound caused by crosstalk.

2. Associating the second location information with the second sound generation virtual object, The method further comprises applying a second HRTF filtering parameterized by the second position information to filter the second multichannel sound associated with the second sound generation virtual object, The method according to claim 1, wherein the second HRTF filtering is configured to equalize the filtered second multichannel sound based on the second position information to compensate for spectral coloration of the filtered second multichannel sound caused by crosstalk.

3. The method according to claim 1, wherein the first position information includes a Y position coordinate.

4. The method according to claim 1, wherein the first position information includes the path direction.

5. The method according to claim 1, wherein the first position information includes the direction of the path from the first sound-generating virtual object to the listener.

6. The method according to claim 1, wherein the first position information includes specifying a region within a virtual sound field.

7. The method according to claim 1, wherein equalization is performed to modify the spatialization provided by the first HRTF filtering.

8. The method according to claim 1, further comprising synthesizing the crosstalk-removed HRTF with the original HRTF in a position-dependent manner.

9. The method according to claim 1, further comprising moving the first sound-generating virtual object as part of video gameplay.

10. The method according to claim 1, wherein application comprises bilinear interpolation based on the first position information.

11. A system comprising at least one processor configured to perform an operation, wherein the operation is Associating the first position information with the first sound generation virtual object, The method comprises applying a first HRTF filtering parameterized by the first position information to filter the first multichannel sound associated with the first sound generation virtual object, The application of the above-mentioned system comprises equalizing the filtered first multichannel sound based on the first position information to compensate for spectral coloration of the filtered first multichannel sound caused by crosstalk.

12. The operation described above is transparent Associating the second location information with the second sound generation virtual object, The method comprises applying a second HRTF filtering parameterized by the second position information to filter the second multichannel sound associated with the second sound generation virtual object, The system according to claim 11, further comprising applying a second HRTF filtering to equalize the filtered second multichannel sound based on the second position information to compensate for spectral coloration of the filtered second multichannel sound caused by crosstalk.

13. The system according to claim 11, wherein the first position information includes a Y position coordinate.

14. The system according to claim 11, wherein the first position information includes the path direction.

15. The system according to claim 11, wherein the first position information includes the direction of the path from the first sound-generating virtual object to the listener.

16. The system according to claim 11, wherein the first position information includes specifying a region within a virtual sound field.

17. The system according to claim 11, comprising equalizing to modify the spatialization provided by the first HRTF filtering.

18. The system according to claim 11, further comprising the operation of synthesizing the crosstalk-deselected HRTF with the original HRTF in a position-dependent manner.

19. The system according to claim 11, wherein the operation further comprises moving the first sound-generating virtual object as part of video gameplay.

20. The system according to claim 11, wherein application comprises bilinear interpolation based on the first position information.

Images