Virtual auditory display filter and associated systems, methods, and non-temporary computer-readable media

Abstract

The exemplary method comprises the step of receiving an audio signal associated with a virtual auditory space location. A digital filter is selected based on the virtual auditory space location. The digital filter includes one or more notch filters containing one or more center frequencies, based on a roughly sigmoid distribution of center frequencies as a function of the virtual auditory space location. The notch filters are configured to produce one or more notches in the frequency spectrum of the audio signal when applied to the audio signal. The digital filters are applied to the audio signal to obtain a processed audio signal. An output audio signal is generated based on the processed audio signal. The output audio signal is provided to the device to produce a virtual auditory display sound.

Description

Technical Field

[0001] The present technology generally relates to virtual auditory display filters, and more particularly to generating virtual auditory display filters, applying virtual auditory display filters to audio signals to generate virtual auditory display sounds in a virtual auditory space, and applications related to virtual auditory display filters.

Background Art

[0002] A three-dimensional (3D) sound system can be implemented by arranging a plurality of speakers in a space, thereby enabling sound to reach from different directions. Headphones, headsets, and earbuds (collectively headphones) are often used to listen to music or other audio. Headphones can simulate 3D sound using a head-related transfer function (HRTF). The HRTF can be a compressed representation of how sound waves interact with the human head and ears. More generally, the HRTF can be used to simulate the effect of sound waves traveling through 3D space.

Summary of the Invention

[0003] In some embodiments, the techniques described herein are one or more non-temporary computer-readable media comprising executable instructions that cause the system to perform a method when executed by one or more processors of the system, the method comprising: generating one or more first digital filters for each of a plurality of virtual auditory space locations, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies based on a first substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, the one or more first notch filters, when applied to a first audio signal, are configured to produce one or more first notches in a first frequency spectrum of the first audio signal based on the one or more first center frequencies; and generating one or more second digital filters for each of the plurality of virtual auditory space locations. Steps include: the one or more second digital filters comprising one or more second notch filters comprising one or more second center frequencies, the one or more second center frequencies being based on a second substantially sigmoid distribution of center frequencies as a function of virtual auditory space locations, the one or more second notch filters being configured, when applied to a second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies; receiving an audio signal, the audio signal having one or more audio sub-signals, the audio sub-signals being associated with virtual auditory space locations; for each of the one or more audio sub-signals: selecting a specific one or more first digital filter and a specific one or more second digital filter based on the virtual auditory space locations associated with the audio sub-signal;Steps include: applying one or more of the specified first digital filters to the audio sub-signal to obtain a first processed audio sub-signal; applying one or more of the specified second digital filters to the audio sub-signal to obtain a second processed audio sub-signal; generating a first output audio signal for a first device based on a plurality of first processed audio sub-signals; generating a second output audio signal for a second device based on a plurality of second processed audio sub-signals; and; The present invention relates to one or more non-temporary computer-readable media, comprising the steps of providing the first output audio signal to the first device and the second output audio signal to the second device.

[0004] In some embodiments, the techniques described herein relate to one or more non-temporary computer-readable media, wherein the virtual auditory space location is a first virtual auditory space location, and the method further comprises: receiving the user's head orientation; and determining a second virtual auditory space location for each of the one or more audio subsignals, based on the first virtual auditory space location and the head orientation associated with the audio subsignal; and selecting a particular one or more first digital filter and a particular one or more second digital filter based on the virtual auditory space location associated with the audio subsignal, or selecting a particular one or more first digital filter and a particular one or more second digital filter based on the second virtual auditory space location.

[0005] In some embodiments, the techniques described herein relate to one or more non-temporary computer-readable media, wherein the particular one or more first digital filters are the first particular one or more first digital filters, the particular one or more second digital filters are the first particular one or more second digital filters, the user's head orientation is the user's first head orientation, and the method: receiving a personalization audio signal associated with a third virtual auditory space location; selecting the second particular one or more first digital filters and the second particular one or more second digital filters based on the third virtual auditory space location; and applying the second particular one or more first digital filters to the personalization audio signal to produce a first processed personalization audio Steps include: acquiring a signal; applying one or more of the second specific second digital filters to the personalization audio signal to acquire a second processed personalization audio signal; generating a third output audio signal for the first device based on the first processed personalization audio signal; generating a fourth output audio signal for the second device based on the second processed personalization audio signal; providing the third output audio signal to the first device and the fourth output audio signal to the second device; receiving the user's second head orientation; determining a fourth virtual auditory space location based on the second head orientation; determining the delta between the third virtual auditory space location and the fourth virtual auditory space location; and The process further comprises modifying the one or more first digital filters and the one or more second digital filters based on the delta.

[0006] In some embodiments, the techniques described herein relating to one or more non-transient computer-readable media, the step of modifying the one or more first digital filters and the one or more second digital filters based on the delta includes the step of modifying the one or more first center frequencies on which the one or more first notch filters are based and the one or more second center frequencies on which the one or more second notch filters are based.

[0007] In some embodiments, the techniques described herein relate to one or more non-transient computer-readable media, the method further comprising the step of generating a first notch mask and a second notch mask using one or more image processing algorithms, wherein the first notch mask specifies a first gain modifier as a function of a virtual auditory space location, and the second notch mask specifies a second gain modifier as a function of a virtual auditory space location; the one or more first notch filters include the one or more first center frequencies and a first gain such that it is modified by the first gain modifier; and the one or more first notch filters include the first O When applied to an audio signal, the one or more second notch filters are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain, the one or more second notch filters include the one or more second center frequencies and a second gain such that it is modified by the second gain modifier, and the one or more second notch filters are configured, when applied to the second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain.

[0008] In some embodiments, the techniques described herein relate to one or more non-transient computer-readable media, wherein the one or more image processing algorithms include one or more of the following: Gaussian functions, sharpening functions, contrast adjustment functions, color correction functions, thresholding functions, edge detection functions, and segmentation functions.

[0009] In some embodiments, the techniques described herein relate to one or more non-temporary computer-readable media, wherein the method further comprises: receiving a selection of an acoustic environment; and determining a first acoustic environment digital filter and a second acoustic environment digital filter based on the acoustic environment; and for each of the one or more audio sub-signals of the one or more audio sub-signals, applying the particular one or more first digital filters to the audio sub-signal to obtain the first processed audio sub-signal, which includes applying the particular one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal; and applying the particular one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal, which includes applying the particular one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

[0010] In some embodiments, the techniques described herein relate to one or more non-transient computer-readable media, wherein the acoustic environment is represented by one or more ambisonic arrays, and the step of determining the first acoustic environment digital filter and the second acoustic environment digital filter based on the acoustic environment includes the step of determining the first acoustic environment digital filter and the second acoustic environment digital filter based on the one or more ambisonic arrays.

[0011] In some embodiments, the techniques described herein relate to one or more non-transient computer-readable media, wherein the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

[0012] In some embodiments, the techniques described herein relate to one or more non-temporary computer-readable media, wherein the first device includes a first ear-mounted device, and the second device includes a second ear-mounted device.

[0013] In some embodiments, the techniques described herein are systems comprising at least one processor and at least one memory, wherein the at least one memory, when executed by the at least one processor, is used in the system: For each of the multiple virtual auditory space locations, one or more first digital filters are generated, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies being based on a first substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, the one or more first notch filters being configured, when applied to a first audio signal, to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies; for each of the multiple virtual auditory space locations, one or more second digital filters are generated, the one or more second digital filters comprising one or more second notch filters, the one or more second notch filters comprising one or more second center frequencies, the one or more second center frequencies being a function of the virtual auditory space locations and The first and second notch filters are based on a second substantially sigmoid distribution of center frequencies, and the first and second notch filters, when applied to a second audio signal, are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the first and second center frequencies; the first and second notch filters are based on a second notch distribution ofThe present invention relates to a system having executable instructions for generating a first output audio signal for a first device based on a plurality of first processed audio sub-signals; generating a second output audio signal for a second device based on a plurality of second processed audio sub-signals; and providing the first output audio signal to the first device and the second output audio signal to the second device.

[0014] In some embodiments, the techniques described herein relate to a system, wherein the virtual auditory space location is a first virtual auditory space location, and the executable instruction, when executed by the at least one processor, causes the system to: receive the user's head orientation; and for each of the one or more audio subsignals, to further determine a second virtual auditory space location based on the first virtual auditory space location and the head orientation associated with the audio subsignal; selecting a particular one or more first digital filter based on the virtual auditory space location associated with the audio subsignal includes selecting a particular one or more first digital filter based on the second virtual auditory space location; and selecting a particular one or more second digital filter based on the virtual auditory space location associated with the audio subsignal includes selecting a particular one or more second digital filter based on the second virtual auditory space location.

[0015] In some embodiments, the techniques described herein relate to a system, wherein the one or more first digital filters are first one or more first digital filters, the one or more second digital filters are first one or more second digital filters, the particular one or more first digital filters are first particular one or more first digital filters, the particular one or more second digital filters are first particular one or more second digital filters, the head orientation is first head orientation, the audio signal having one or more audio sub-signals is first audio signal having first one or more audio sub-signals, and the executable instruction, when executed by the at least one processor, causes the system to: receive a personalization audio signal having a third virtual auditory space location; and based on the third virtual auditory space location, a second particular one or more first digital filter and a second particular one or more second digital filter Selecting; applying the second specific one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; applying the second specific one or more second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; generating a third output audio signal for the first device based on the first processed personalization audio signal; generating a fourth output audio signal for the second device based on the second processed personalization audio signal; providing the third output audio signal to the first device and the fourth output audio signal to the second device; receiving the user's second head orientation; determining a fourth virtual auditory space location based on the second head orientation; determining the delta between the third virtual auditory space location and the fourth virtual auditory space location;Furthermore, based on the delta, the system is to select one or more second first digital filters and one or more second second digital filters, the second one or more first digital filters and the second one or more second second digital filters being for use while receiving a second input audio signal having one or more second audio sub-signals.

[0016] In some embodiments, the techniques described herein relate to a system, wherein the executable instruction, when executed by the at least one processor, causes the system to further generate a first notch mask and a second notch mask using one or more image processing algorithms, wherein the first notch mask specifies a first gain modifier based on the virtual auditory space location, and the second notch mask specifies a second gain modifier based on the virtual auditory space location: the one or more first notch filters are generated using the one or more first center frequencies based on the first substantially sigmoid distribution of the center frequencies as a function of the virtual auditory space location, and a first gain such that it is modified by the first gain modifier, and the one or more first notches The filters, when applied to the first audio signal, are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain, the one or more second notch filters are generated using the one or more second center frequencies based on the second substantially sigmoid distribution of the center frequencies as a function of virtual auditory space location, and the second gain as modified by the second gain modifier, the one or more second notch filters, when applied to the second audio signal, are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain.

[0017] In some embodiments, the techniques described herein relate to a system, wherein the executable instruction, when executed by the at least one processor, causes the system to: receive a selection of an acoustic environment; and, based on the acoustic environment, determine a first acoustic environment digital filter and a second acoustic environment digital filter; and for each of the one or more audio sub-signals, applying the particular one or more first digital filters to the audio sub-signal to obtain the first processed audio sub-signal includes applying the particular one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal; and applying the particular one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal includes applying the particular one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

[0018] In some embodiments, the techniques described herein relate to a system in which the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

[0019] In some embodiments, the techniques described herein relate to a system, wherein the first device includes a first ear-mounted device, and the second device includes a second ear-mounted device.

[0020] In some embodiments, the technique described herein is a method comprising: generating a first virtual auditory display filter, the first virtual auditory display filter comprising a first set of first functions, one or more of which, when applied to a first audio signal having a first location in a virtual auditory space, generate a first processed audio signal having a first frequency response having one or more first notches at one or more first center frequencies based on the first location, the one or more first notches having one or more first peak-to-trough depths of up to -10 dB; generating a second virtual auditory display filter, the second virtual auditory display filter comprising a second set of second functions, one or more of which, when applied to the first audio signal, have one or more second notches at one or more second center frequencies based on the first location The method comprises the steps of: generating a second processed audio signal having a frequency response of 2, wherein one or more second notches have one or more second peak-to-trough depths of up to -10 dB; receiving the second audio signal having a second location in the virtual auditory space; applying the first virtual auditory display filter, which includes a first subset of a first function selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response; applying the second virtual auditory display filter, which includes a second subset of a second function selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response; providing the third processed audio signal to a first sound output device; and providing the fourth processed audio signal to a second sound output device.

[0021] In some embodiments, the techniques described herein relate to methods wherein the one or more first center frequencies are based on a first substantially sigmoid distribution of center frequencies as a function of location in the virtual auditory space, and the one or more second center frequencies are based on a second substantially sigmoid distribution of center frequencies as a function of location in the virtual auditory space.

[0022] In some embodiments, the techniques described herein, relating to a method, further comprising the steps of receiving the user's head orientation: applying a first virtual auditory display filter, which includes the first subset of a first function selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response; applying the first virtual auditory display filter, which includes the third subset of a first function selected based on the second location and the head orientation, to the second audio signal to generate the third processed audio signal having a third frequency response; applying the second virtual auditory display filter, which includes the second subset of a second function selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response; applying the second virtual auditory display filter, which includes the fourth subset of a second function selected based on the second location and the head orientation, to the second audio signal to generate the fourth processed audio signal having a fourth frequency response.

[0023] In some embodiments, the techniques described herein relate to methods comprising the steps of: generating a first notch mask and a second notch mask using one or more image processing algorithms; the first notch mask specifying a first depth modifier as a function of location in the virtual auditory space; the second notch mask specifying a second depth modifier as a function of location in the virtual auditory space; modifying the one or more first peak-to-trough depths based on the first depth modifier; and modifying the one or more second peak-to-trough depths based on the second depth modifier.

[0024] In some embodiments, the techniques described herein relate to methods, wherein the one or more image processing algorithms include one or more of the following: Gaussian functions, sharpening functions, contrast adjustment functions, color correction functions, thresholding functions, edge detection functions, and segmentation functions.

[0025] In some embodiments, the techniques described herein relate to methods, wherein the first set of first functions comprises a first infinite impulse response digital filter, and the second set of second functions comprises a second infinite impulse response digital filter.

[0026] In some embodiments, the technique described herein is a method comprising: receiving a set of a plurality of first digital filters, one or more first digital filters being generated for each of a plurality of virtual auditory space locations, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies being based on a first substantially sigmoid distribution of center frequencies as a function of virtual auditory space locations, the one or more first notch filters being configured, when applied to a first audio signal, to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies; receiving a set of a plurality of second digital filters, one or more second digital filters being generated for each of a plurality of virtual auditory space locations, the one or more first The two digital filters include one or more second notch filters, each including one or more second center frequencies, the one or more second center frequencies being based on a second substantially sigmoid distribution of center frequencies as a function of virtual auditory space locations, and the one or more second notch filters are configured, when applied to a second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies; the steps of: receiving a personalization audio signal having a virtual auditory space location; selecting a particular one or more first digital filter and a particular one or more second digital filter based on the virtual auditory space location; and applying the particular one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; Applying the specific one or more second digital filters to the personalized audio signal to obtain a second processed personalized audio signal; providing a first output audio signal based on the first processed personalized audio signal to a first device and a second output audio signal based on the second processed personalized audio signal to a second device; receiving user perception of the first sound output by the first device and the second sound output by the second device; and modifying the set of the plurality of first digital filters and the set of the plurality of second digital filters based on the user perception.

[0027] In some aspects, the techniques described herein relate to a method, wherein the virtual auditory space location is a first virtual auditory space location, and the step of modifying the set of the plurality of first digital filters and the set of the plurality of second digital filters based on the user perception includes: determining a second virtual auditory space location based on the user perception; determining a delta between the first virtual auditory space location and the second virtual auditory space location; and modifying the set of the plurality of first digital filters and the set of the plurality of second digital filters based on the delta.

[0028] In some aspects, the techniques described herein relate to a method, wherein the step of receiving the user perception includes receiving the user's head orientation, and the step of determining the second virtual auditory space location based on the user perception includes determining the second virtual auditory space location based on the user's head orientation.

[0029] In some aspects, the techniques described herein relate to a method, wherein the step of receiving the user perception includes receiving one or more gestures of the user, and the step of determining the second virtual auditory space location based on the user perception includes determining the second virtual auditory space location based on one or more gestures of the user.

[0030] In some aspects, the techniques described herein relate to a method, wherein the set of multiple first digital filters is the first set of multiple first digital filters, the set of multiple second digital filters is the first set of multiple second digital filters, and the step of modifying the set of multiple first digital filters based on the user perception includes selecting a second set of multiple first digital filters based on the user perception, and the step of modifying the set of multiple second digital filters based on the user perception includes selecting a second set of multiple second digital filters based on the user perception.

[0031] In some aspects, the techniques described herein relate to a method, wherein the step of modifying the set of multiple first digital filters based on the user perception includes modifying the one or more first center frequencies, and the step of modifying the set of multiple second digital filters based on the user perception includes modifying the one or more second center frequencies.

[0032] In some aspects, the techniques described herein relate to a method, further comprising determining a spatialization accuracy estimate based on the user perception; and providing the spatialization accuracy estimate.

Brief Description of the Drawings

[0033] [Figure 1] It is a diagram of an environment in which a virtual auditory display system and a virtual auditory display device can operate in some embodiments.

[0034] [Figure 2-1] Figure 2A is a block diagram illustrating the components of a virtual auditory display system in several embodiments. Figure 2B is a block diagram illustrating the components of an ear-worn device in several embodiments.

[0035]

[0036] [Figure 2-2] Figure 2C is a block diagram illustrating the process for generating an acoustic environment digital filter in several embodiments. Figure 2D is a block diagram illustrating the operation of the spatialization engine of a virtual auditory display system in several embodiments.

[0037]

[0038] [Figure 3A-1] This is a block diagram of a method for generating and applying a digital filter in several embodiments. [Figure 3A-2] This is a block diagram of a method for generating and applying a digital filter in several embodiments, and is a continuation of Figure 3A-1.

[0039] [Figure 3B] This is a block illustrating the components of a filter generation system in several embodiments.

[0040] [Figure 4-1] Figure 4A is a graph of the frequency response of a digital audio signal in several embodiments. [Figure 4-2] Figures 4B and 4C are graphs of the frequency response of digital audio signals in several embodiments.

[0041] [Figure 5]Figure 5A shows the distribution of center frequencies as a function of azimuth (x-axis) and elevation (y-axis) for the left ear. Figure 5B shows the distribution of center frequencies as a function of azimuth (x-axis) and elevation (y-axis) for the right ear.

[0042]

[0043] [Figure 6] Figure 6A is a graph of the center frequency of a digital filter as a function of the elevation angle relative to the head azimuth, according to several embodiments. Figure 6B is a graph of user experience data from multiple trials using five different digital filters, varying as a function of the notch center frequency, according to several embodiments.

[0044]

[0045] [Figure 7-1] Figures 7A, 7B, 7C, and 7D illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments. [Figure 7-2] Figures 7E, 7F, 7G, and 7H illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments. [Figure 7-3] Figures 7I, 7J, 7K, and 7L illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments. [Figure 7-4] Figures 7M, 7N, 7O, and 7P illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments. [Figure 7-5] Figures 7Q, 7R, 7S, and 7T illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments. [Figure 7-6]Figures 7U, 7V, 7W, and 7X illustrate parameter modifier masks that may be applied to modify the gain of a digital filter in several embodiments.

[0046] [Figure 8A] This figure illustrates the head shadow gain produced by a digital filter in several embodiments. [Figure 8B] This figure illustrates the head shadow gain produced by a digital filter in several embodiments.

[0047] [Figure 8-1] Figure 8C is a diagram illustrating the output of a digital filter applied to a digital audio signal according to several embodiments. Figure 8D is a diagram illustrating user experience data for transfer functions based on digital filters according to several embodiments, and user experience data for transfer functions of prior art.

[0048]

[0049] [Figure 8-2] Figure 8E is a diagram illustrating an example head-related transfer function (HRTF).

[0050] [Figure 9A] This figure illustrates a method for generating a digital filter according to several embodiments. [Figure 9B] This figure illustrates a method for generating a digital filter according to several embodiments.

[0051] [Figure 10A] This diagram illustrates a method for applying a digital filter according to several embodiments. [Figure 10B] This diagram illustrates a method for applying a digital filter according to several embodiments.

[0052] [Figure 10C] This figure illustrates a method for generating and applying a virtual auditory display filter in several embodiments.

[0053] [Figure 11] Figures 11A and 11B depict exemplary user interfaces for displaying a virtual audio display representation in several embodiments. Figure 11C depicts exemplary user interfaces for adjusting settings for the virtual audio display in several embodiments.

[0054]

[0055] [Figure 12] These are several images illustrating exemplary use cases of display filter technology in several embodiments.

[0056] [Figure 13] Figures 13A and 13B illustrate methods for personalizing digital filters in several embodiments.

[0057] [Figure 14A] This diagram illustrates how to personalize a digital filter in several embodiments. [Figure 14B] This diagram illustrates how to personalize a digital filter in several embodiments.

[0058] [Figure 15-1] Figures 15A, 15B, and 15C depict exemplary user interfaces for calibrating a virtual auditory display device in several embodiments. Figures 15D, 15E, and 15F depict exemplary user interfaces for personalizing the virtual auditory display of a virtual auditory display device in several embodiments.

[0059]

[0060] [Figure 15-2] Figures 15G, 15H, 15I, and 15J depict exemplary user interfaces for providing information regarding the calibration of a virtual auditory display device and the personalization of the virtual auditory display of the virtual auditory display device in several embodiments.

[0061] [Figure 16] This is a block diagram of an exemplary digital device in several embodiments.

[0062] Throughout the drawings, similar reference numerals are understood to refer to the same parts, components, and structures. [Modes for carrying out the invention]

[0063] HRTF can be for a single person. Generating an individual HRTF typically requires highly specialized environments and acoustic testing equipment. The person must remain still in an anechoic chamber for approximately 30 minutes while audio signals are emitted from different known locations. Microphones are placed at each of the person's ears to capture the audio signals. However, this method presents challenges because spurious responses may exist due to factors such as the chamber, audio signal source, and microphones, which need to be eliminated in order to obtain an accurate Head Related Impulse Response (HRIR) that can later be converted into an HRTF. Furthermore, any movement by the person can affect the measurement, which can result in an inaccurate HRTF for that person. Another practical limitation of measuring HRIR is that the time to collect is directly proportional to the number of discrete coordinates, which practically limits the resolution of the resulting HRTF.

[0064] To overcome the disadvantages of individual HRTFs, so-called universal HRTFs are used. Such universal HRTFs can be produced by averaging or otherwise combining measurements from multiple individuals. However, such combinations typically result in the loss of the individual characteristics necessary to produce accurate virtual 3D sound for each person. As a result, such universal HRTFs may not accurately localize sounds in the virtual 3D space for all users, particularly sounds located directly in front of the user at approximately zero azimuth and zero elevation angles. Figure 8E depicts an exemplary HRTF 810.

[0065] Another conventional approach attempts to simulate personalized HRTFs using photogrammetry of the head, torso, and auricle, or other methods employing highly precise scanning of the head, torso, and auricle via time-of-flight or structured light. A physical acoustic model is then generated based on the resulting scanned form. However, this approach may not yield a credible rendering of the virtual 3D space because, after the physical scan is measured, the physics-based simulation of sound interacting with the modeled surface can introduce complexity and inaccuracies into the resulting physical acoustic cues.

[0066] The techniques described herein provide technical solutions to the technical problems of the conventional methods described above. These techniques may utilize virtual auditory display filters that can produce accurately rendered sounds at their respective locations in a virtual auditory space. The virtual auditory display filters may utilize spectral shaping techniques, employing equalizers, filters, and / or dynamic range compression to manipulate the frequency spectrum of the audio signal. The virtual auditory display filters may be generated without relying on direct physical measurements (e.g., measurements in an anechoic chamber, photogrammetry, etc.).

[0067] A virtual auditory display filter is a function that manipulates the frequency spectrum of an audio signal, or may include such a function. A virtual auditory display filter is a digital filter, such as a parametric equalization (EQ) filter, that allows adjustment of parameters such as center frequency, gain, quality (Q or q), cutoff frequency, gradient, bandwidth, and / or filter type, or may include such a digital filter. These parameters may be set as a function of the location of the sound in the virtual auditory space. A function or digital filter may affect the frequency spectrum of an audio signal by creating notches and peaks in the audio signal. Notches, peaks, and other spectral shaping of the audio signal precisely place the resulting sound in the virtual auditory space. Furthermore, notches, peaks, and other spectral shaping of the audio signal can produce a processed audio signal that, in the example of a music recording, can be used to output a high-quality, clear sound that accurately represents the original recorded performance and allows the listener to perceive the nuances and subtleties of the original recorded performance. As described herein, a digital filter may refer to a digital filter, a function, and / or any combination of one or more functions or one or more digital filters.

[0068] A virtual auditory space can be described as a person's virtual 3D sound environment in which the person can perceive sounds as originating from any location in that virtual 3D sound environment. In the described technology, each location in the virtual auditory space may have an associated function or digital filter applied to the audio signal having that location. The application of the function or digital filter to the audio signal having a location produces a sound that the person perceives as coming from that location, which may be called a virtual auditory display sound. Thus, a person who may be wearing headphones, earbuds, or other in-ear devices may experience a virtual auditory display sound. Other advantages of the described technology will become apparent.

[0069] Figure 1 shows an environment 150 in which a virtual auditory display system and a virtual auditory display device interfaced with the virtual auditory display system may operate in several embodiments. As depicted, the environment 150 includes a virtual auditory display system 102 and a virtual auditory display device 100. Both the virtual auditory display system 102 and the virtual auditory display device 100 may include a system. Both the virtual auditory display system 102 and the virtual auditory display device 100 may render sound in a virtual auditory space for the wearer of the virtual auditory display device 100.

[0070] The virtual auditory display system 102 may include a binauralizer 138. The binauralizer may include a system memory 118, which may include a left ear digital filter map 120a and a right ear digital filter map 120b. The binauralizer 138 may include a left ear convolution engine 116a, a right ear convolution engine 116b, and a spatialization engine 114. The virtual auditory display system 102 may include other components, modules, and / or engines, such as those described with reference to Figure 2A.

[0071] In some embodiments, the virtual auditory display system 102 may be, or include, a software application that can be run on a digital device. The digital device is any device having at least one processor and memory. Digital devices are discussed further herein with reference to, for example, Figure 16. For example, the virtual auditory display system 102 may be a software application that runs on a general-purpose computing device such as a laptop or desktop computer. As another example, the virtual auditory display system 102 may be a software application that runs on a mobile device such as a telephone or tablet. In other embodiments, the virtual auditory display system 102 may be, or include, a software application or firmware application that runs on a dedicated computing device such as the virtual auditory display device 100.

[0072] The virtual auditory display device 100 may include a first ear-mounted device 102a and a second ear-mounted device 102b. The first ear-mounted device 102a and the second ear-mounted device 102b may each be any ear-mounted, ear-attached, or near-ear device, such as one earphone from a pair of earphones, one earbud from a pair of earbuds, headphones from a headset, speakers from a virtual reality headset, and so on. In some embodiments, the virtual auditory display device 100 is included in the concurrently pending U.S. Patent Application No. 1, filed on the same day as this application and titled “VIRTUAL AUDITORY DISPLAY DEVICES AND ASSOCIATED SYSTEMS, METHODS, AND DEVICES” mentioned above. This may be one embodiment of a virtual auditory display device as described in the reference. The first ear-mounted device 102a and / or the second ear-mounted device 102b may include components such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, and / or a magnetometer that detect the head orientation of the wearer wearing the first ear-mounted device 102a and the second ear-mounted device 102b.

[0073] In some embodiments, a digital device (e.g., a laptop or desktop computer) may receive an encoded audio file 106 having one or more audio channels. Examples of encoded audio files 106 include 2.0 (2-channel audio), 2.1 (3-channel audio), 5.1 (6-channel audio), 7.1.4 (12-channel audio), and 9.1.6 (16-channel audio). The digital device may decode the encoded audio file 106 to obtain a decoded audio object 108 and an input audio signal 112 containing one or more audio subsignals (alternatively, audio channels). Each of the decoded audio object 108 and / or audio subsignals may have associated coordinates that identify the location of the audio object in a virtual auditory space. The coordinates may be Cartesian, spherical, and / or polar coordinates. While specific examples of encoded audio files are described herein, the technique is not limited to such examples and may be used with audio files having any number of channels.

[0074] The digital device may transmit coordinates 110 to the spatialization engine 114 and the input audio signal 112 to the left ear convolution engine 116a and the right ear convolution engine 116b. In some embodiments, the virtual auditory display system 102 receives the encoded audio file 106, decodes the encoded audio file 106, and obtains the decoded audio object 108 and the input audio signal 112.

[0075] For example, as illustrated with reference to Figures 11A and 11B, the user interface component of the virtual auditory display system 102 may provide a user interface that allows the user to select an acoustic environment. The spatialization engine 114 may receive a selection 134 of an acoustic environment 132 from the wearer via the user interface component and use the selection 134 to process audio signals to be transmitted to the first ear-mounted device 102a and the second ear-mounted device 102b of the virtual auditory display device 100.

[0076] For example, as illustrated with reference to Figures 15A to 15J, the user interface component of the virtual auditory display system 102 may provide a user interface 128 that enables the user to perform a calibration and / or personalization procedure 136 to calibrate and / or personalize the virtual auditory display system 102. The wearer may use the user interface 128 to personalize the virtual auditory display system 102 so that the user's perception of the sound's location matches the sound's location in the virtual auditory space. The user-perceived location of the sound may be transmitted to the spatialization engine 114 in signal 130.

[0077] The spatialization engine 114 may determine a first acoustic environment digital filter and a second acoustic environment digital filter based on the acoustic environment 132. The acoustic environment digital filter is or may include a digital filter applied to an audio signal to manipulate the audio signal to produce the effects of audio being reproduced, generated, or produced in a particular acoustic environment. The spatialization engine 114 may provide the first acoustic environment digital filter to the left ear convolution engine 116a and the second acoustic environment digital filter to the right ear convolution engine 116b.

[0078] While the virtual auditory display system 102 is receiving the input audio signal 112, one or both of the first ear-mounted device 102a and the second ear-mounted device 102b may detect the head orientation of the wearer of the virtual auditory display device 100 and provide the head orientation and the audio source distance 126 (which may be specified by the wearer) to the virtual auditory display system 102.

[0079] The binauralizer 138 may acquire a plurality of first processed audio subsignals and a plurality of second processed audio subsignals for each of the one or more audio subsignals. The binauralizer 138 may do this by determining a specific first location in the virtual auditory space for the audio subsignal based on the virtual auditory space location and head orientation associated with the audio subsignal. The left ear digital filter map 120a maps the location in the virtual auditory space to a digital filter and / or function for the first ear-mounted device 102a, and the right ear digital filter map 120b maps the location in the virtual auditory space to a digital filter and / or function for the second ear-mounted device 102b.

[0080] The virtual auditory display filter is a function and / or digital filter that the virtual auditory display system 102 applies to the audio signal to create a virtual auditory display sound, or may include such a function and / or digital filter. For example, the generation system, which is discussed in more detail with reference to Figures 3A and 3B, may generate a virtual auditory display filter that the virtual auditory display system 102 applies to the audio signal.

[0081] The binauralizer 138 may select a specific first digital filter and / or function from the left ear digital filter map 120a and a specific second digital filter and / or function from the right ear digital filter map 120b in the system memory 118. The binauralizer 138 may provide the specific first digital filter and / or function to the left ear convolution engine 116a and the specific second digital filter and / or function to the right ear convolution engine 116b.

[0082] The left ear convolution engine 116a may apply a specific first digital filter and / or function and a first acoustic environment digital filter to the audio sub-signal to obtain a first processed audio sub-signal. The left ear convolution engine 116a may then generate an output audio signal 122a for the first ear-mounted device 102a based on a plurality of first processed audio sub-signals. The right ear convolution engine 116b may apply a specific second digital filter and / or function and a second acoustic environment digital filter to the audio sub-signal to obtain a second processed audio sub-signal. The right ear convolution engine 116b may then generate an output audio signal 122b for the second ear-mounted device 102b based on a plurality of second processed audio sub-signals. Graph 124a depicts an exemplary impulse response for the output audio signal 122a, and graph 124b depicts an exemplary impulse response for the output audio signal 122b.

[0083] Figure 2A is a block diagram illustrating the components of a virtual auditory display system 102 in several embodiments. The virtual auditory display system 102 may include a binauralizer 138, a communication module 202, an audio input module 204, an audio output module 206, a calibration and personalization module 208, a user interface module 210, and data storage 220.

[0084] The communication module 202 may transmit requests and / or data between the components of the virtual auditory display system 102 and any other components or devices such as the virtual auditory display device 100 and the generation system 380 (as described with reference to, for example, Figures 3A and 3B). The communication module 202 may also receive requests and / or data between the components of the virtual auditory display system 102 and any other components or devices.

[0085] The audio input module 204 may receive an input audio signal 112 from, for example, a general-purpose computing device on which a virtual auditory display system 102 is running. The audio output module 206 may provide an output audio signal 122a to the first ear-mounted device 102a and an output audio signal 122b to the second ear-mounted device 102b.

[0086] The calibration and personalization module 208 may calibrate the IMU and / or other sensors of the first ear-mounted device 102a and the second ear-mounted device 102b. The calibration and personalization module 208 may generate a personalization audio signal and receive personalization information for personalizing filters. The user interface module 210 may provide a user interface that allows the user to, among other things, select an acoustic environment, select an audio visualization, control the audio volume, and request that calibration and / or personalization procedures be performed by the virtual auditory display system 102.

[0087] The data storage 220 may include data that is stored, accessed, and / or modified by any of the engines, components, modules, or similar of the virtual auditory display system 102. The data storage 220 may include any number of data storage structures, such as tables, databases, lists, and / or similar. The data storage 220 may include data that is stored in memory (e.g., random access memory: RAM), on disk, or in any combination of memory and disk.

[0088] Figure 2B is a block diagram illustrating the components of a first ear-mounted device 102a and a second ear-mounted device 102b in several embodiments. The first ear-mounted device 102a may include a memory 250, an IMU sensor system 252 (inertial measurement unit sensor system), a magnetometer 254, a microcontroller 256, a power management component 258, an audio DSP 260 (audio digital signal processor), a microphone 262, and a speaker 264. The second ear-mounted device 102b may include a memory 250, an IMU sensor system 252 (inertial measurement unit sensor system), a magnetometer 254, an audio DSP 260, a microphone 262, and a speaker 264.

[0089] Memory 250 may store software and / or firmware. The IMU sensor system 252 and / or magnetometer 254 may detect the head orientation of the wearer of the virtual auditory display device 100 and / or user interaction with the virtual auditory display device 100. The microcontroller 256 may execute software and / or firmware stored in memory 250 or in the storage of the microcontroller 256.

[0090] The power management component 258 may provide power management. The audio DSP 260 may process audio signals and perform functions such as noise cancellation. The microphone 262 may capture audio such as ambient audio and / or audio from the wearer of the first ear-mounted device 102a. The speaker 264 may output sound based on output audio signals 122a and 122b.

[0091] The first ear-mounted device 102a and / or the second ear-mounted device 102b may include components other than those depicted in Figure 2B, such as switches, interconnects, and oscillators. The first ear-mounted device 102a may be a primary device, and the second ear-mounted device 102b may be a secondary device. Therefore, the second ear-mounted device 102b does not have to include the microcontroller 256. In some embodiments, the second ear-mounted device 102b includes the microcontroller 256.

[0092] The engines, components, modules, or similar components of the virtual auditory display system 102, the first ear-mounted device 102a, the second ear-mounted device 102b, or the generation system 380 (described, for example, with reference to Figure 3B) may be hardware, software, firmware, or any combination thereof. For example, each engine, component, module, or similar component may include functions, software, instructions maintained in memory, and / or any combination thereof, executed by dedicated hardware (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or similar). The software and / or firmware may be executed by one or more processors.

[0093] While a limited number of engines, components, and modules are depicted in Figures 2A, 2B, and 3B, any number of engines, components, and modules or similar may exist. Furthermore, individual engines, components, and modules may perform any number of functions, including functions of multiple modules as described herein. Moreover, while the virtual auditory display system 102, the first ear-mounted device 102a, the second ear-mounted device 102b, and the generation system 380 may be depicted as having a single engine, component, or module from among several, the virtual auditory display system 102, the first ear-mounted device 102a, the second ear-mounted device 102b, and the generation system 380 may have multiple engines, components, modules or similar that perform specific functions. For example, while the first ear-mounted device 102a is depicted as having a single audio DSP 260, the first ear-mounted device 102a may include multiple audio DSPs 260.

[0094] Figure 3A is a block diagram of a method 300 for personalizing, generating, and applying digital filters in several embodiments. A generation system 380 (see Figure 3B) may perform the generation of digital filters (steps 304 to 310), and a virtual auditory display system 102 may perform the personalization of digital filters (step 302) and the application of digital filters (steps 312 to 314).

[0095] The digital filter may be one or more parametric equalization (EQ) filters that allow adjustment of parameters such as center frequency, gain, quality (Q or q), cutoff frequency, gradient, bandwidth, and / or filter type, or may include such filters. The parametric EQ filter may be a bi-quadratic filter, or may include such filters. The bi-quadratic filter may be a peaking, low-shelf, and high-shelf filter, or may include such filters. In some embodiments, the digital filter may be one or more finite impulse response (FIR) filters, or may include such filters. The FIR filter may be generated from or based on one or more infinite impulse response (IIR) filters. In some embodiments, the digital filter may be one or more IIR filters, or any other suitable type of digital filter, or may include such filters.

[0096] The digital filters generated by the generation system 380 may be organized into multiple groups. These groups may include groups of notch filters, shadow filters, shelf filters, peak filters, beam filters, stereo filters, rear filters, top filters, top transition filters, bottom filters, bottom transition filters, and broadside filters. Other groups are possible. A particular digital filter or group of digital filters may be used to define the location of a sound in a virtual auditory space (e.g., a group of notch filters). A particular digital filter or group of digital filters may be used to ensure that a sound meets required thresholds for tone quality, clarity, brightness, and similar properties.

[0097] A digital filter may consist of an algorithm and one or more parameters for the algorithm, or include these. For example, the algorithm may be a high-shelf, low-shelf, and peaking algorithm, or include these. One or more parameters may be a center frequency, quality (Q or q), gain, and sampling frequency, or include these. For example, a notch digital filter may specify a peaking algorithm, an initial center frequency of 6600 Hz, a Q of 15, and an initial gain of -85 decibels (dB). One or more parameters may be modified. For example, the initial center frequency may be shifted to obtain a shifted center frequency, and the initial gain may be modified by a parameter modifier (see, for example, the discussion relating to Figures 7A to 7X), and a coefficient having any value such as a value between 0 and 1 (including both ends). A digital filter may consist of one or more parameters that are modified, or include these.

[0098] Digital filters may be generated and used based on how the digital filter represents how an individual's geometry interacts with sound waves. For example, a digital filter with a high-shelf algorithm may produce a high shelf, which could be a virtual representation of how an individual's conchaecular cavity geometry interacts with sound waves.

[0099] Method 300 may include a calibration and / or personalization step 302. The virtual auditory display system 102 (e.g., the calibration and personalization module 208) may perform calibration of the IMU and / or other sensors of the virtual auditory display device 100 using various devices and / or services 326, such as one or both of the first ear-mounted device 102a and the second ear-mounted device 102b, a cloud-based computing service, and / or peripherals to computing devices such as a camera.

[0100] The virtual auditory display system 102 (e.g., the calibration and personalization module 208) may perform personalization of the virtual auditory display system using various methodologies and / or techniques such as 1) perception of user-instructed actions and / or acoustic cues 316; 2) acoustic quality user feedback 318; 3) anatomical measurements 320; 4) demographic information 322; and 5) audiometry measurements 324.

[0101] The perception 316 of user-instructed actions and / or acoustic cues may include capturing the user's response to the location of the acoustic cue. The response may be the user's voice response captured using the computing device's microphone, the user's gestures (e.g., head and / or arm movements) captured using the computing device's camera and / or one or both of the first ear-mounted device 102a and the second ear-mounted device 102b, and user input captured via the computing device's graphical user interface (GUI).

[0102] The sound quality user feedback 318 may include user instruction feedback on sound quality (e.g., responses to questions regarding quality metrics such as brightness, warmth, clarity, etc., and responses to questions provided by a GUI or audio user interface (AUI)), and observations of user behavior such as the user's preferences for music and / or notifications via a GUI or AUI.

[0103] The anatomical measurements 320 may include measurements of the user's anatomical features, such as the head, auricle, and / or concha, via scanning or prediction. The anatomical measurements 320 for one or more users may include direct measurements (e.g., via silicone ear impressions) and indirect measurements obtained via sensors and / or computer peripherals.

[0104] Demographic information 322 may include information provided by the user, such as the user's age or other demographic values, and a digital fingerprint of the user generated from one or more user characteristics, such as age, gender, and / or other user characteristics.

[0105] The hearing test measurements 324 may include those provided by user input and / or those obtained through acoustic measurements, such as those taken in an anechoic chamber while audio signals are emitted from different known locations.

[0106] Method 300 may include a step 304 in which a generation system 380 (for example, a model generation module 386 of the generation system 380, see Figure 3B) generates, modifies, and / or receives a plurality of models 370. The plurality of models 370 may include one or more outer ear models 354, which may include one or more auricle models 356 and one or more concha models 358. The plurality of models 370 may include one or more head and torso models 360 and one or more ear canal models 362. The generation system 380 may generate the plurality of models 370 based on calibration and / or personalization information obtained in step 302.

[0107] For each of the multiple models 370, and for each location in the virtual auditory space, the generation system 380 may generate one or more first digital filters (for the left ear) and one or more second digital filters (for the right ear) based on the model. Therefore, for each of the multiple models 370, the generation system 380 may generate multiple first digital filters and multiple second digital filters for each location in the virtual auditory space.

[0108] For example, for one or more head and torso models 360, the generation system 380 may generate one or more first digital filters and one or more second digital filters that take into account shoulder width and / or spread, head diameter, neck height, and other factors. For one or more conchae models 358, the generation system 380 may generate one or more first digital filters and one or more second digital filters that represent the acoustic effects of the physical features of the conchae. These features include, but are not limited to, the depth, width, and angle of the conchae.

[0109] As another example, for one or more auricle models 356, the generation system 380 may generate one or more first digital filters and one or more second digital filters that represent the acoustic effects of the physical features of the auricle. These features include, but are not limited to, the height, width, depth, location on the head, and flare angle relative to the head of the auricle. For one or more ear canal models 362, the generation system 380 may generate one or more first digital filters and one or more second digital filters that take into account the physical proportions of the auricle, conchae, and other parts of the ear.

[0110] Similarly, in step 304, for each location in the virtual auditory space, the generating system 380 may sum, aggregate, or otherwise combine a plurality of first digital filters to form a combined first digital filter, and may sum, aggregate, or otherwise combine a plurality of second digital filters to form a combined second digital filter. The combined first digital filter may be one or more finite impulse response (FIR) filters, or include them. The combined second digital filter may also be one or more FIR filters, or include them. Thus, in the result of step 304, for all locations in the virtual auditory space, there may be a set of combined first digital filters and a set of combined second digital filters.

[0111] In step 306, the generation system 380 may generate a mapping or association of the combined first digital filters to their corresponding locations in the virtual auditory space for the left ear. The generation system 380 may also generate a mapping or association of the combined second digital filters to their corresponding locations in the virtual auditory space for the right ear. The generation system 380 may utilize Cartesian, polar, and / or spherical polar coordinates for the mapping or association.

[0112] In step 308, the generation system 380 may generate a file, database, or other data structure that includes a mapping or association of the combined first digital filters to their corresponding locations in the virtual auditory space and a mapping or association of the combined second digital filters to their corresponding locations in the virtual auditory space.

[0113] In step 310, the generation system 380 may provide or store files, databases, or other data structures on one or more non-temporary computer-readable media of the devices. The devices may be the first ear-mounted device 102a and / or the second ear-mounted device 102b, a mobile device, such as a telephone or tablet, a laptop or desktop computer, another device, or any combination thereof.

[0114] In step 312, the virtual auditory display system 102 may select a first digital filter and a second digital filter combined for use. After selection, in step 314, the virtual auditory display system 102 may utilize the first digital filter and the second digital filter combined in various applications such as music rendering. Various applications of the disclosed technology are discussed, for example, with reference to Figure 12.

[0115] In some embodiments, in step 304, for each of the multiple models 370, the generation system 380 may generate one or more first digital filters and one or more second digital filters for each combination of azimuth and elevation angles at locations in the virtual auditory space that are 1 meter (1 m) away from a center point representing a virtual listener in the virtual auditory space, in increments of 1 degree. An increment of azimuth is approximately -180 degrees (inclusive) to approximately +180 degrees (inclusive). An increment of elevation is approximately -90 degrees (inclusive) to approximately +90 degrees (inclusive). Thus, there are 65,160 combinations of azimuth and elevation angles, and therefore 65,160 locations in the virtual auditory space, each location being 1 m away from the center point. Therefore, the generation system 380 may generate 65,160 sets of one or more first digital filters and 65,160 sets of one or more second digital filters.

[0116] In some embodiments, method 300 may include a step in which the generation system 380 reduces the number of locations in the virtual auditory space for which the digital filters are generated or stored. For example, after step 304, the generation system 380 may select a true subset from a combined first set of digital filters and a true subset from a combined second set of digital filters.

[0117] In an embodiment where there are 65,160 locations in the virtual auditory space, the generation system 380 may select a true subset from the set of combined first digital filters that includes approximately 7,000, for example, 7,220, combined first digital filters. Similarly, the generation system 380 may select a true subset from the set of combined second digital filters that includes approximately 7,000, for example, 7,220, combined second digital filters.

[0118] The generation system 380 may select a proper subset that adequately represents locations in the virtual auditory space while reducing the amount of storage required for the set of digital filters and the amount of time spent selecting and processing the digital filters. The generation system 380 may achieve these objectives by other means, for example, generating or storing mappings or associations for the reduced number of locations in the virtual auditory space.

[0119] In some embodiments, in step 304, the generation system 380 does not sum, aggregate, or otherwise combine a first digital filter formed by combining a plurality of first digital filters, or a second digital filter formed by combining a plurality of second digital filters. Thus, in the result of step 304, there may be a set of plurality of first digital filters and a set of plurality of second digital filters for all locations in the virtual auditory space. A proper subset of the set of plurality of first digital filters and a proper subset of the set of plurality of second digital filters may be used as described herein.

[0120] In such embodiments, in step 306, the generation system 380 may instead generate a mapping or association of a plurality of first digital filters to their corresponding locations in the virtual auditory space for the left ear, and a mapping or association of a plurality of second digital filters to their corresponding locations in the virtual auditory space for the right ear.

[0121] Furthermore, in such embodiments, in step 308, the generation system 380 may instead generate a file, database, or other data structure that includes mappings or associations of a plurality of first digital filters to their corresponding locations in the virtual auditory space, and mappings or associations of a plurality of combined second digital filters to their corresponding locations in the virtual auditory space.

[0122] In some embodiments, the generation system 380 generates multiple sets of digital filters for locations in the virtual auditory space. The generation system 380 may generate a first set of digital filters for the left ear and a first set of digital filters for the right ear, as described herein. The generation system 380 may then generate one or more second sets of digital filters for the left ear and one or more second sets of digital filters for the right ear, based on the first set of digital filters for the left ear and the first set of digital filters for the right ear. Each pair of sets may be for different archetypes representing different user populations or groupings of users.

[0123] The generation system 380 may generate a second set of one or more digital filters for the left ear and a second set of one or more digital filters for the right ear by modifying one or more parameters of the digital filters for the left ear and the digital filters for the right ear. For example, the generation system 380 may modify the center frequencies of the notch filters included in the first set of digital filters for the left ear and the first set of digital filters for the right ear. The generation system 380 may personalize the digital filters for the user by modifying the center frequencies of the notch filters, as illustrated with reference to Figures 15A to 15F, for example. The generation system 380 may do this to adjust the delta between the actual location of the sound in the virtual auditory space and the location of the sound as perceived by the wearer.

[0124] In some embodiments, the generation system 380 may generate a first set of digital filters for the left ear and a first set of digital filters for the right ear for distances of 1 m from a center point representing a virtual listener in the virtual auditory space, as described herein. The generation system 380 may generate one or more second sets of digital filters for the left ear and one or more second sets of digital filters for the right ear for other distances from the center point. The generation system 380 may generate one or more second sets of digital filters for the left ear based on the first set of digital filters for the left ear, and one or more second sets of digital filters for the right ear based on the first set of digital filters for the right ear. For example, the generation system 380 may increase the gain of the digital filters for distances closer than 1 m from the center point and decrease the gain of the digital filters for distances further than 1 m from the center point. Other methods will become apparent.

[0125] Figure 3B is a block illustrating the components of the generation system 380 in several embodiments. The generation system 380 may include a communication module 382, ​​a filter generation module 384, a model generation module 386, a parameter generation module 388, a parameter mask module 390, a digital filter adjustment module 392, a user interface module 394, and data storage 396.

[0126] The communication module 382 may transmit requests and / or data between the components of the generation system 380 and any other system, component, or device, such as the virtual auditory display system 102. The communication module 382 may also receive requests and / or data between the components of the generation system 380 and any other system, component, or device.

[0127] The filter generation module 384 may generate digital filters and acoustic environment digital filters. The filters may consist of one or more algorithms and, optionally, one or more parameters for one or more algorithms, or include these.

[0128] The model generation module 386 may generate multiple models, modify them, or access them. The parameter generation module 388 may generate parameters for a digital filter.

[0129] The parameter mask module 390 may generate a parameter modifier mask. The parameter mask module 390 may use image processing techniques to generate the parameter modifier mask. The parameter mask module 390 may use the parameter modifier mask to determine one or more parameter modifiers for one or more parameters of the filter. The parameter mask module 390 may use one or more parameter modifiers to modify one or more parameters.

[0130] The digital filter adjustment module 392 may receive parameters for the digital filter from the user and modify the digital filter based on the received parameters. The user interface module 394 may provide a user interface that allows the user to, among other things, listen to the sound output from the audio signal produced by the application of the digital filter and modify the parameters of the digital filter.

[0131] The data storage 396 may include data that is stored, accessed, and / or modified by any of the engines, components, modules, or similar of the generation system 380. The data storage 396 may include any number of data storage structures, such as tables, databases, lists, and / or similar. The data storage 396 may include data that is stored in memory (e.g., random access memory (RAM)), on disk, or in any combination of memory and disk.

[0132] Figures 4A to 4C are graphs of the frequency response of digital audio signals in several embodiments. Figure 4A is a graph of the frequency response for three audio signals. Each audio signal has a notch at a different center frequency. The center frequency of the notch is a factor in specifying the location of the sound corresponding to the audio signal in the virtual auditory space, which means where the user (e.g., the wearer of the first ear-mounted device 102a and the second ear-mounted device 102b) perceives the location of the sound.

[0133] The first audio signal is for a first sound having a first location in the virtual auditory space at a distance of 1 meter (m), an azimuth angle of 0 degrees (0°), and an elevation angle of 0 degrees (0°). The second audio signal is for a second sound having a second location in the virtual auditory space at a distance of 1 m, an azimuth angle of 5 degrees (5°), and an elevation angle of 0 degrees (0°). The third audio signal is for a third sound having a third location in the virtual auditory space at a distance of 1 m, an azimuth angle of 10 degrees (10°), and an elevation angle of 0 degrees (0°). Figure 4A shows the variation in the center frequency notches of the three signals due to the difference in location in the virtual auditory space. The virtual auditory display system 102 applies notch filters to the three audio signals to produce a notch in each of the three frequency responses in order to produce three sounds at specified locations in the virtual auditory space. A notch filter is or may include a parametric EQ filter, the parameters of which are the center frequency, gain, and bandwidth.

[0134] Figure 4B is a graph 420 of the frequency response of an audio signal to which a digital filter is applied according to several embodiments. The frequency response has three notches at three different center frequencies. The audio signal is for a sound having a location in a virtual auditory space at a distance of 1 meter (m), an azimuth angle of 0 degrees (0°), and an elevation angle of 0 degrees (0°). The virtual auditory display system 102 applies three notch filters to produce three notches in the frequency response of the audio signal. The notch filters are or may include parametric EQ filters, the parameters being center frequency, gain, and bandwidth.

[0135] Figure 4C is a graph of the frequency responses of two audio signals to which digital filters are applied according to several embodiments. Each frequency response has notches at different center frequencies. The virtual auditory display system 102 applies three notch filters to produce three notches in each frequency response. The notch filters are or may include parametric EQ filters, the parameters of which are center frequency, gain, and bandwidth.

[0136] In the examples depicted in Figures 4A to 4C, the peak-to-trough decibel values ​​for the azimuth angle between -10° and 95°, and the elevation angle between -30° and 45°, reach less than -30 dB. If the virtual sound source is located within the proposed azimuth / elevation boundary, peak-to-trough decibel values ​​of -30 dB or higher may be beneficial for producing accurate sound source localization for the listener.

[0137] Figure 5A illustrates the distribution of center frequencies 500 as a function of azimuth (x-axis) and elevation (y-axis) for the left ear. Figure 5B illustrates the distribution of center frequencies 550 as a function of azimuth (x-axis) and elevation (y-axis) for the right ear. Distributions 500 and 550 show that for any given azimuth angle, the center frequencies follow a roughly sigmoid curve or have a roughly sigmoid shape or distribution. Similarly, for any given elevation angle, the center frequencies follow a roughly sigmoid curve or have a roughly sigmoid shape or distribution.

[0138] For example, Figure 6A is graph 600 of the center frequency curve 602 as a function of elevation angle (x-axis) for the right ear with an azimuth angle of zero degrees (0°). The center frequency values ​​range from approximately 4900 Hz to approximately 8700 Hz for elevation angles from -90 degrees to 90 degrees. The center frequency curve 602 has a roughly sigmoid shape or distribution.

[0139] Returning to Figures 5A and 5B, the virtual auditory display system 102 may use distributions 500 and / or 550 to determine the center frequencies for one or more notches in the frequency spectrum of the audio signal. The virtual auditory display system 102 may determine the center frequencies for one or more notches based on the location of the sound in the virtual auditory space that the audio signal will produce on the first ear-mounted device 102a and the second ear-mounted device 102b.

[0140] That is, based on the location of the sound in the virtual auditory space (for example, specified by azimuth and elevation angles), the virtual auditory display system 102 may determine the center frequencies of one or more notches in the frequency spectrum of the audio signal that causes the first ear-mounted device 102a and the second ear-mounted device 102b to produce sound. The virtual auditory display system 102 may determine the center frequency of the first notch in the frequency spectrum of the audio signal by accessing distributions 500 and 550. The virtual auditory display system 102 may determine the center frequencies of the second and subsequent notches in the frequency spectrum of the audio signal based on distributions 500 and 550, and one or more shifts from the center frequencies obtained from distributions 500 and 550.

[0141] In some embodiments, in addition to or as an alternative to utilizing distributions 500 and / or 550, the virtual auditory display system 102 may utilize one or more center frequency curves, each of which may be for different azimuth or elevation values, such as the center frequency curve 602 in Figure 6A. The virtual auditory display system 102 determines the center frequencies for one or more notches in the frequency spectrum of the audio signal based on the location of the resulting sound in the virtual auditory space.

[0142] Figure 6B is a graph of user experience data from multiple trials using five different digital filters, varying as a function of the notch center frequency, in several embodiments. Points 652a, 652b, 652c, 652d, and 652e are the averages of 15 user trials, totaling 75 user trials, from which real-time user feedback on perceived sound locations in the virtual auditory space was collected. Bars 654a, 654b, 654c, 654d, and 654e each represent a + or -1 standard deviation. Points 652b, 652c, 652d, and 652e demonstrate that there is an observed delta of approximately 2.5° for every 150Hz added to the notch center frequency. Line 656 can be fitted to point 652. The virtual auditory display system 102 may use the linear function that produced line 656 to determine the center frequencies to be used for one or more notches based on the elevation angle of the sound produced. For example, for a particular range of elevation angles (e.g., between approximately zero and approximately 50 degrees, or between approximately 10 and approximately 40 degrees), the virtual auditory display system 102 may use the linear function that produced line 656 to determine one or more shifts from the center frequencies in the range. The virtual auditory display system 102 may do this in addition to, or as an alternative to, using the distributions 500 and / or distributions 550 depicted in Figures 5A and 5B.

[0143] Figures 7A to 7X depict parameter modifier masks that can be applied to modify the parameters used when generating digital filters in several embodiments. For a notch filter, Figure 7A depicts the right ear parameter modifier mask 700a, and Figure 7B depicts the left ear parameter modifier mask 700b. For a head shadow filter, Figure 7C depicts the right ear parameter modifier mask 705a, and Figure 7D depicts the left ear parameter modifier mask 705b. For a shelf filter, Figure 7E depicts the right ear parameter modifier mask 710a, and Figure 7F depicts the left ear parameter modifier mask 710b. For a peak filter, Figure 7G depicts the right ear parameter modifier mask 715a, and Figure 7H depicts the left ear parameter modifier mask 715b. Regarding the beam filter, Figure 7I depicts the right ear parameter modifier mask 720a, and Figure 7J depicts the left ear parameter modifier mask 720b. Regarding the stereo filter, Figure 7K depicts the right ear parameter modifier mask 725a, and Figure 7L depicts the left ear parameter modifier mask 725b.

[0144] Regarding the rear filter, Figure 7M depicts the right ear parameter modifier mask 730a, and Figure 7N depicts the left ear parameter modifier mask 730b. Regarding the top filter, Figure 7O depicts the right ear parameter modifier mask 735a, and Figure 7P depicts the left ear parameter modifier mask 735b. Regarding the top transition filter, Figure 7Q depicts the right ear parameter modifier mask 740a, and Figure 7R depicts the left ear parameter modifier mask 740b. Regarding the bottom filter, Figure 7S depicts the right ear parameter modifier mask 745a, and Figure 7T depicts the left ear parameter modifier mask 745b. Regarding the bottom transition filter, Figure 7U depicts the right ear parameter modifier mask 750a, and Figure 7V depicts the left ear parameter modifier mask 750b. Regarding the broadside filter, Figure 7W depicts the right ear parameter modifier mask 755a, and Figure 7X depicts the left ear parameter modifier mask 755b.

[0145] The parameter modifier masks depicted in Figures 7A to 7X specify parameter modifier values ​​as functions of location in virtual auditory space, such as by azimuth and elevation. The parameter modifier values ​​may be a range between any two values. In some embodiments, the parameter modifier values ​​range between zero (0) (including this value) and other values ​​such as 1, 0.2, 0.4, 0.8 (including this value). The parameter mask module 390 may generate a parameter modifier mask by specifying a particular region in virtual auditory space where the value will be 1, and by specifying that regions other than that particular region will have a value of zero (0). For example, for the right ear parameter modifier mask 700a in Figure 7A, the particular region in virtual auditory space may be an azimuth angle of approximately -50 degrees to approximately 110 degrees, and an elevation angle of approximately -70 degrees to approximately 30 degrees. The parameter mask module 390 may use other specific regions for the parameter modifier mask 700a and other parameter modifier masks in Figures 7A to 7X.

[0146] The parameter mask module 390 may use an image processing algorithm to create a continuous transition of values ​​between a specific region and other regions, thereby generating a parameter modifier mask using parameter modifier values. For example, the parameter mask module 390 may use an image processing algorithm such as a Gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and / or a segmentation function. In some embodiments, the parameter mask module 390 uses a Gaussian blur mask to generate parameter modifier values. The parameter mask module 390 may generate a parameter modifier mask for the right ear, and then obtain a parameter modifier mask for the right ear by reflecting the parameter modifier mask for the right ear with respect to the vertical axis at a zero (0) azimuth value.

[0147] The filter generation module 384 may use the parameter modifier masks depicted in Figures 7A to 7X to select one or more parameter modifiers that can be used to modify one or more parameters to obtain one or more modified parameters, based on the location of the sound in the virtual auditory space. For example, the filter generation module 384 may use one or more parameter modifiers to modify the gain of the digital filter. In some embodiments, the filter generation module 384 obtains one or more modified parameters by multiplying one or more parameters by one or more parameter modifiers. Other uses of parameter modifiers will become apparent.

[0148] According to several embodiments, Figure 8A depicts the gain distribution 870 for the head shadow for the left ear, and Figure 8B depicts the gain distribution 880 for the head shadow for the right ear. Gain distribution 870 depicts how the gain changes as a sound source transition from generally location 872 for the right ear, generally location 874 in front of the wearer, and generally location 876 for the left ear. Gain distribution 880 depicts how the gain changes as a sound source transition from generally location 882 for the left ear, generally location 884 in front of the wearer, and generally location 886 for the left ear.

[0149] Figure 8C depicts the gain distribution 860 of an application of a digital filter to an audio signal according to one embodiment. The gain distribution 890 shows several notches 864 across the head shadow 862. The several notches 864 have center frequencies between 10^3 Hz and 10^4 Hz.

[0150] Figure 8D depicts user experience data for digital filters and for conventional head-related transfer functions (HRTFs) according to several embodiments. Conventional HRTFs have been used as a standard model for many past and present HRTF applications. Panel 800 in Figure 8D reports the difference between the user-perceived elevation angle of a virtual sound object and the actual elevation angle of the sound object for both digital filters and conventional HRTFs for 150 trials. For the digital filter trials, point 804 is the average user-perceived elevation angle, and band 802 is the standard deviation of the user-perceived elevation angle. For the HRTF trials, point 808 is the average user-perceived elevation angle, and band 806 is the standard deviation of the user-perceived elevation angle.

[0151] Panel 850 in Figure 8D reports the difference between the user-perceived azimuth of a virtual sound object and the actual azimuth of the sound object for both a digital filter and a conventional HRTF for 150 trials. For the digital filter trials, point 852 is the average user-perceived azimuth, and band 854 is the standard deviation of the user-perceived azimuth. For the HRTF trials, point 858 is the average user-perceived azimuth, and band 856 is the standard deviation of the user-perceived azimuth.

[0152] For elevation trials, the closer the elevation delta is to 0°, the more accurate the representation of the virtual sound object becomes. Similarly, for azimuth trials, the closer the elevation delta is to 0°, the more accurate the representation of the virtual sound object becomes. User experience data shows that the digital filter improves the elevation delta from an average of approximately 30.19° with a standard deviation of approximately 12.54° to an average of approximately -0.03° with a standard deviation of approximately 4.12°. User experience data also shows that the digital filter improves the azimuth delta from an average of approximately -0.64° with a standard deviation of approximately 7.76° to an average of approximately -0.02° with a standard deviation of approximately 2.04°. The data shows that digital filters, in several embodiments, improve the accuracy and precision of the virtual sound object.

[0153] Figure 9A illustrates a method 900 for generating digital filters according to several embodiments. A generation system 380 may perform method 900. The generation system 380 may perform method 900 to generate a set of combined first digital filters and a set of combined second digital filters. Method 900 begins in step 902, where the generation system 380 (e.g., filter generation module 384) may generate a roughly sigmoid distribution of center frequencies for the right ear (see, for example, Figure 5A) and a roughly sigmoid distribution of center frequencies for the left ear (see, for example, Figure 5B).

[0154] In step 904, the generation system 380 (e.g., parameter mask module 390) generates a parameter modifier mask for the right ear and a parameter modifier mask for the left ear (see, for example, Figures 7A to 7X). The generation system 380 may generate the parameter modifier masks using one or more image processing algorithms. One or more image processing algorithms may include one or more of the following: a Gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and a segmentation function.

[0155] In step 906, for each of the multiple locations in the virtual auditory space, the generation system 380 (e.g., parameter generation module 388) may generate one or more first parameters for one or more first digital filters and one or more second parameters for one or more second digital filters. One or more first parameters may include one or more first q, one or more first gains, and one or more first center frequencies. One or more second parameters may include one or more second q, one or more second gains, and one or more second center frequencies.

[0156] The generation system 380 may use parameter modifier masks for the right ear and parameter modifier masks for the left ear to select one or more parameter modifiers that the generation system 380 can use to modify one or more parameters to obtain one or more modified parameters, based on the location in the virtual auditory space. In some embodiments, the generation system 380 obtains one or more modified parameters by multiplying one or more parameters by one or more parameter modifiers.

[0157] The generation system 380 may determine one or more center frequencies for one or more notches in the frequency spectrum of the audio signal for the right ear and the audio signal for the left ear by utilizing the approximate sigmoid distribution of center frequencies for the right ear and / or the approximate sigmoid distribution of center frequencies for the left ear. The generation system 380 may determine the center frequencies for one or more notches based on their location in the virtual auditory space.

[0158] In step 908, for each location, the generation system 380 (e.g., filter generation module 384) may generate one or more first digital filters, including one or more first notch filters, each containing one or more first parameters. The generation system 380 may generate one or more first notch filters using one or more first qs, one or more first gains, and one or more first center frequencies. The one or more first notch filters are configured to produce one or more first notches in the first frequency spectrum of the first audio signal, according to one or more first qs, one or more first gains, and one or more first center frequencies, when the generation system 380 applies one or more notch filters to the audio signal for the right ear.

[0159] In step 910, for each location, the generation system 380 (e.g., filter generation module 384) may generate one or more combined first digital filters for that location based on one or more first digital filters. In some embodiments, one or more first digital filters are IIR filters, and one or more combined first digital filters are FIR filters.

[0160] In step 912, for each location, the generation system 380 may store one or more combined first digital filters associated with that location (for example, in the data storage 220).

[0161] In step 914, for each location, the generation system 380 (e.g., filter generation module 384) may generate one or more second digital filters, including one or more second notch filters with one or more second parameters. The generation system 380 may generate one or more second notch filters using one or more second qs, one or more second gains, and one or more second center frequencies. The one or more second notch filters are configured to produce one or more second notches in the second frequency spectrum of the second audio signal, according to one or more second qs, one or more second gains, and one or more second center frequencies, when the generation system 380 applies one or more notch filters to the audio signal for the left ear.

[0162] In step 916, for each location, the generation system 380 (e.g., filter generation module 384) may generate one or more combined second digital filters for that location based on one or more second digital filters. In some embodiments, one or more second digital filters are IIR filters, and one or more combined second digital filters are FIR filters.

[0163] In step 918, for each location, the generation system 380 may store one or more combined second digital filters associated with that location (for example, in the data storage 220).

[0164] In step 920, the generating system 380 tests whether there are more locations for which the generating system 380 will generate digital filters. If there are, method 900 returns to step 906. The generating system 380 may perform method 900 multiple times to generate multiple sets of combined first digital filters and multiple sets of combined second digital filters. Each pair of sets of digital filters may be for a different archetype.

[0165] In some embodiments, the generation system 380 may perform method 900 several times to generate multiple sets of combined first digital filters and combined second digital filters. Each pair in the set may be for a different archetype representing a different user population or grouping of users.

[0166] Figure 9B illustrates a method 950 for generating digital filters in several embodiments. A generation system 380 may perform method 900. Method 950 includes certain steps that may be generally similar to certain steps of method 900. The generation system 380 (e.g., various components of the generation system 380) may perform method 950. The generation system 380 may perform method 950 to generate a first set of digital filters and a second set of digital filters.

[0167] Method 950 begins in step 952, where the generation system 380 (e.g., parameter generation module 388) may generate a first approximate sigmoid distribution of center frequencies and a second approximate sigmoid distribution of center frequencies. In step 954, the generation system 380 (e.g., parameter mask module 390) may generate a first parameter modifier mask and a second parameter modifier mask.

[0168] In step 956, for each of the multiple virtual auditory space locations, the generation system 380 (e.g., parameter generation module 388) may generate one or more first parameters for one or more first digital filters and one or more second parameters for one or more second digital filters. One or more first parameters may include one or more first q, one or more first gains, and one or more first center frequencies. One or more second parameters may include one or more second q, one or more second gains, and one or more second center frequencies.

[0169] In step 958, for each virtual auditory space location, the generation system 380 may generate one or more first digital filters, each containing one or more first notch filters containing one or more first parameters. Step 958 is generally similar to step 980 of method 900.

[0170] In step 960, for each virtual auditory space location, the generation system 380 may store one or more first digital filters associated with the virtual auditory space location. Step 960 is generally similar to step 912 of method 900.

[0171] In step 962, for each virtual auditory space location, the generation system 380 may generate one or more second digital filters, each containing one or more second notch filters containing one or more second parameters. Step 962 is generally similar to step 914 of method 900.

[0172] In step 964, for each virtual auditory space location, the generation system 380 may store one or more second digital filters associated with the virtual auditory space location. Step 964 is generally similar to step 918 of method 900.

[0173] In step 966, the generation system 380 tests whether there are more virtual auditory space locations for which the generation system 380 will generate digital filters. If there are, method 900 returns to step 956. The generation system 380 may perform method 950 multiple times to generate multiple sets of one or more first digital filters and multiple sets of one or more second digital filters.

[0174] Methods 900 and 950 may include additional steps. For example, the generation system 380 may provide testing of the digital filter. The generation system 380 (e.g., user interface module 394) may provide a user interface that enables playback of the sound produced by the audio signal to which the digital filter is applied. The user may listen to the sound and decide that one or more parameters of the digital filter should be modified. For example, the user may modify the parameters of the digital filter to ensure that the sound meets the required thresholds for tone quality, clarity, brightness, and similar. The generation system 380 may provide another user interface that enables the user to modify one or more parameters of the digital filter. The generation system 380 (e.g., digital filter adjustment module 392) may receive one or more parameters of the digital filter from the user and modify the digital filter based on the received one or more parameters.

[0175] Figure 10A illustrates a method 1000 for applying digital filters according to several embodiments. A virtual auditory display system 102 and a virtual auditory display device 100 may perform method 1000. Method 1000 begins in step 1002, where the virtual auditory display system 102 (e.g., binauralizer 138) receives a combined first set of digital filters and a combined second set of digital filters. In step 1004, the virtual auditory display system 102 (e.g., binauralizer 138) receives an input audio signal containing one or more audio subsignals. Each of the one or more audio subsignals has a location in the virtual auditory space.

[0176] While receiving an input audio signal, the virtual auditory display system 102 performs steps 1006 to 1020 of method 1000. In step 1006, one or both of the first ear-mounted device 102a and the second ear-mounted device 102b detect the head orientation of the user wearing the first ear-mounted device 102a and the second ear-mounted device 102b. The first ear-mounted device 102a and / or the second ear-mounted device 102b provide the head orientation to the virtual auditory display system 102.

[0177] In step 1008, for each of the one or more audio subsignals, the virtual auditory display system 102 determines a specific location in the virtual auditory space based on the location of the audio subsignal and the head orientation. In step 1010, for each audio subsignal, the virtual auditory display system 102 selects a specific combination of one or more first digital filters and a specific combination of one or more second digital filters based on the specific location.

[0178] In step 1012, for each audio sub-signal, the virtual auditory display system 102 applies a specific combination of first digital filters to the audio sub-signal to obtain a first processed audio sub-signal. In step 1014, for each audio sub-signal, the virtual auditory display system 102 applies a specific combination of second digital filters to the audio sub-signal to obtain a second processed audio sub-signal.

[0179] In step 1016, the virtual auditory display system 102 tests whether there are more audio subsignals to process. If there are, method 1000 returns to step 1008. If there are no more, method 1000 proceeds to step 1018. After processing all audio subsignals, the virtual auditory display system 102 obtains a plurality of first processed audio subsignals and a plurality of second processed audio subsignals.

[0180] In step 1018, the virtual auditory display system 102 generates a first output audio signal for the left ear-mounted device based on a plurality of first processed audio sub-signals, and a second output audio signal for the right ear-mounted device based on a plurality of second processed audio sub-signals. The virtual auditory display system 102 provides the first output audio signal to the first ear-mounted device 102a and the second output audio signal to the second ear-mounted device 102b.

[0181] In step 1020, the first ear-mounted device 102a outputs a first sound based on a first output audio signal, and the second ear-mounted device 102b outputs a second sound based on a second output audio signal. In this way, the virtual auditory display system 102 can use method 1000 to provide virtual auditory display sound based on audio signals that may have multiple audio sub-signals (or channels) which would typically require multiple speakers to produce a surround sound effect. The virtual auditory display system 102 can provide virtual auditory display sound to the user using only the first ear-mounted device 102a and the second ear-mounted device 102b.

[0182] Figure 10B illustrates a method 1050 for applying a digital filter according to several embodiments. Method 1050 includes certain steps that may be generally similar to certain steps of Method 1000. A virtual auditory display system 102 may perform Method 1050.

[0183] Method 1050 begins in step 1052, where a virtual auditory display system 102 (e.g., binauralizer 138) receives one or more sets of first digital filters and one or more sets of second digital filters. In step 1054, the virtual auditory display system 102 (e.g., binauralizer 138) receives an audio signal having one or more audio subsignals. Each of the one or more audio subsignals is associated with a virtual auditory spatial location.

[0184] In step 1056, the virtual auditory display system 102 receives the user's head orientation. In step 1058, for each of the one or more audio subsignals, the virtual auditory display system 102 determines a specific virtual auditory space location based on the virtual auditory space location and head orientation. In step 1060, for each audio subsignal, the virtual auditory display system 102 selects a specific one or more first digital filter and a specific one or more second digital filter based on the virtual auditory space location or a specific virtual auditory space location.

[0185] In step 1062, for each audio sub-signal, the virtual auditory display system 102 applies one or more specific first digital filters to the audio sub-signal to obtain a first processed audio sub-signal. In step 1064, for each audio sub-signal, the virtual auditory display system 102 applies one or more specific second digital filters to the audio sub-signal to obtain a second processed audio sub-signal.

[0186] In step 1066, the virtual auditory display system 102 tests whether there are more audio subsignals to process. If there are, method 1050 returns to step 1058. If there are none, method 1000 proceeds to step 1068. After processing all audio subsignals, the virtual auditory display system 102 obtains a plurality of first processed audio subsignals and a plurality of second processed audio subsignals.

[0187] In step 1068, the virtual auditory display system 102 generates a first output audio signal for a first device based on a plurality of first processed audio sub-signals, and a second output audio signal for a second device based on a plurality of second processed audio sub-signals. The first device is, for example, a first ear-mounted device 102a, or may include the same, and the second device is, for example, a second ear-mounted device 102b, or may include the same. In step 1070, the virtual auditory display system 102 provides the first output audio signal to the first device and the second output audio signal to the second device.

[0188] Figure 10C illustrates a method 1080 for generating and applying a virtual auditory display filter in several embodiments. A generation system 380 and a virtual auditory display system 102 may perform method 1000. Method 1080 begins in step 1082, where the generation system 380 (e.g., parameter generation module 388) may generate a first substantially sigmoid distribution of center frequencies and a second substantially sigmoid distribution of center frequencies. In step 1084, the generation system 380 (e.g., parameter mask module 390) may generate a first parameter modifier mask and a second parameter modifier mask, including a first notch parameter modifier mask and a second notch parameter modifier mask.

[0189] In step 1086, the generation system 380 (e.g., filter generation module 384) may generate a first virtual auditory display filter and a second virtual auditory display filter. The first virtual auditory display filter may include a first set of first functions. One or more first functions, when applied to a first audio signal having a first location in the virtual auditory space, may generate a first processed audio signal having a first frequency response having one or more first notches at one or more first center frequencies based on the first location. One or more first notches may have one or more first peak-to-trough depths of up to -10 dB (e.g., approximately -30 dB).

[0190] The second virtual auditory display filter may include a second set of second functions. One or more second functions, when applied to the first audio signal, may produce a second processed audio signal having a second frequency response having one or more second notches at one or more second center frequencies based on a second location. One or more second notches may have one or more second peak-to-trough depths of up to -10 dB (e.g., approximately -30 dB).

[0191] In step 1088, the virtual auditory display system 102 may receive an audio signal having a second location in the virtual auditory space. For example, the virtual auditory display system 102 may receive an audio signal from a digital device running on it. In step 1090, the virtual auditory display system 102 may receive the user's head orientation from, for example, the virtual auditory display device 100 used by the user.

[0192] In step 1092, the virtual auditory display system 102 may apply a first virtual auditory display filter to the second audio signal, which includes a first subset of a first function selected based on a second location, to generate a third processed audio signal having a third frequency response. In step 1094, the virtual auditory display system 102 may apply a second virtual auditory display filter to the second audio signal, which includes a second subset of a second function selected based on a second location, to generate a fourth processed audio signal having a fourth frequency response. In step 1094, the virtual auditory display system 102 may provide the first processed audio signal to a first sound output device (e.g., a first ear-mounted device 102a) and the second processed audio signal to a second sound output device (e.g., a second ear-mounted device 102b).

[0193] The virtual auditory display system 102 may perform steps 1088 to 1094 of method 1080 while the virtual auditory display system 102 is receiving an input audio signal which may correspond to, for example, a music file, an audio stream, a podcast, or any other audio.

[0194] Methods 1000, 1050, and 1080 may include additional steps not shown in Figures 10A to 10C. For example, these methods may include a step in which the virtual auditory display system 102 receives a selection of an acoustic environment, and a step in which the virtual auditory display system 102 determines a first acoustic environment digital filter and a second acoustic environment digital filter based on the acoustic environment. The acoustic environment may be represented by one or more ambisonic arrays. The virtual auditory display system 102 may determine the acoustic environment digital filters based on one or more ambisonic arrays. The virtual auditory display system 102 may apply the digital filters and the acoustic environment digital filters to obtain a processed audio sub-signal. Other modifications to these methods will become apparent.

[0195] Figure 2C is a block diagram illustrating a process 290 for generating acoustic environment digital filters in several embodiments. The first speaker 292a and the second speaker 292b may be located in a specific acoustic environment such as a concert hall, vehicle, nightclub, or similar. The sound output from the first speaker 292a and the second speaker 292b is captured by a microphone 294 and converted into a signal. A virtual auditory display system 102 (e.g., a filter generation module 384) generates one or more ambisonic digital filters 296 based on the signal. One or more ambisonic digital filters 296 are a set of one or more acoustic environment digital filters 298.

[0196] Figure 2D is a block diagram illustrating the operation of the spatialization engine 270 of the virtual auditory display system 102 in several embodiments. The spatialization engine 270 may be part of or a separate component of the binauralizer 138. In block 272, the spatialization engine 270 receives the acoustic environment selected by the user interface (UI), and in block 274, determines an acoustic environment digital filter based on the selected acoustic environment. In block 276, the spatialization engine 270 receives the coordinates of the decoded audio object and the input audio signal. In block 278, the spatialization engine 270 matches the index of the acoustic environment digital filter to the input audio signal index. In block 280, the spatialization engine 270 applies a convolution matrix to the output coordinates and input audio signal of block 278.

[0197] In block 282, the spatialization engine 270 receives the user head orientation and audio source distance signals. In block 284, the spatialization engine performs ambisonics-to-binaural conversion based on the output of block 280 and the user head orientation and audio source distance signals by applying a digital filter to the audio signal received in block 276, based on the location of the audio signal in the virtual auditory space. In block 286, the spatialization engine 270 outputs an audio signal for the left ear, and in block 288, the spatialization engine 270 outputs an audio signal for the right ear.

[0198] Figures 11A and 11B depict exemplary user interfaces 1100 for displaying a virtual audio display representation in several embodiments. A virtual auditory display system 102 (e.g., user interface module 210) may provide the user interface 1100. Although Figures 11A and 11B are described with reference to a virtual auditory display device 100, a virtual auditory display system 102 may provide the user interface 1100 for other devices.

[0199] The user interface 1100 includes an icon 1104 labeled "VAD" indicating that the virtual auditory display device 100 is connected to the virtual auditory display system 102, and an icon 1102 labeled "IMU" indicating that the IMU-based sensor system of the virtual auditory display device 100 is calibrated. The user interface 1100 may also include an encoded representation dropdown 1114 that allows the wearer to select how the virtual auditory display system 102 should represent the audio received by the virtual auditory display system 102. Exemplary encoded representations are mono (single audio channel), stereo (two channels of audio), 5.1 (six channels of audio), 7.1 (eight channels of audio), 7.1.4 (twelve channels of audio), and 9.1.6 (sixteen channels of audio).

[0200] The user interface 1100 also includes an acoustic environment dropdown 1116 that allows the wearer to select an acoustic environment in which the virtual auditory display system 102 should render the virtual auditory display. Examples of acoustic environments include a dry acoustic environment, a studio acoustic environment, a car acoustic environment, a telephone acoustic environment, a club acoustic environment, and a headphone acoustic environment. Based on the selected acoustic environment, the virtual auditory display system 102 may select an acoustic environment digital filter and apply it together with the virtual auditory display filter. The virtual auditory display sound will sound different to the wearer based on the selected acoustic environment. The user interface 1100 also includes an output volume slider 1122 that allows the wearer to adjust the volume of the sound output from the virtual auditory display device 100.

[0201] The user interface 1100 also includes a representation 1108 of a virtual audio display. In Figure 11A, the virtual auditory display system 102 depicts representation 1108 as a virtual sphere surrounding the head 1112, which represents the wearer's head from a top-rear viewpoint. The user interface 1100 also displays sounds in their corresponding locations in the virtual auditory space relative to the wearer's head, relative to the head 1112 on representation 1108. For example, sound 1110a is depicted as pointing to the left, down, and rear of the head 1112, because the actual sounds corresponding to sound 1110a have those locations in the virtual auditory space. Sound 1110b is depicted as pointing to the top, behind, and slightly left of the head 1112, sound 1110c is depicted as pointing to the front and right of the head 1112, and sound 1110d is depicted as pointing to the right, down, and rear of the head 1112.

[0202] While outputting sound, the virtual auditory display device 100 detects the wearer's head orientation and transmits the head orientation to the virtual auditory display system 102. The virtual auditory display system 102 updates the representation 1108 based on the detected head orientation. The virtual auditory display system 102 may move the head 1112 and the sound 1110 based on the detected head orientation.

[0203] The user interface 1100 also includes a virtual auditory display representation dropdown 1118 that allows the wearer to select how the virtual auditory display system 102 provides virtual auditory display representations. Examples of virtual auditory display representations include a custom representation that provides the wearer with a top-right rear viewpoint of representation 1108 (as depicted in Figure 11A), a top representation that provides the wearer with a top viewpoint of representation 1108 (as depicted in Figure 11B) (from the top of the sphere in Figure 11A), and a back representation that provides the wearer with a rear viewpoint of representation 1108 (from the rear of the sphere in Figure 11A).

[0204] The user interface 1100 also includes a location button 1120 that, when selected by the wearer, can change the representation 1108 of the virtual auditory display system 102 so that a location specified by specific coordinates (e.g., zero-degree azimuth, zero-degree elevation) can be directly in front of the head 1112. The user interface 1100 also includes a settings icon 1106 that, when selected by the wearer, can cause the virtual auditory display system 102 to provide an exemplary user interface for adjusting settings for the virtual audio display.

[0205] Figure 11C depicts an exemplary user interface 1150 for adjusting settings for a virtual audio display in several embodiments. A virtual auditory display system 102 (e.g., user interface module 210) may provide the user interface 1150. The user interface 1150 includes an icon 1152 labeled "IMU" indicating that the IMU-based sensor system of the virtual auditory display device 100 is calibrated. The user interface 1150 also includes a button 1154 labeled "Recalibrate" which allows the wearer to choose to have the virtual auditory display system 102 perform a calibration and / or personalization process.

[0206] The user interface 1150 also includes an icon 1156 labeled "VAD" indicating that the virtual auditory display device 100 is connected to the virtual auditory display system 102, a recommendation for a virtual auditory display filter 1158, and a button 1160 labeled "Personalize" in which the wearer can choose to have the virtual auditory display system 102 perform the personalization part of the calibration and / or personalization process. The user interface 1150 also shows an estimate of the wearer's spatial accuracy of the virtual auditory display, and a button 1162 labeled "Test" in which the wearer can choose to have the virtual auditory display system 102 provide a test procedure for the wearer, which allows the wearer to test whether the wearer can accurately localize the virtual auditory display sounds.

[0207] The user interface 1150 also includes a group 1164 of icons (labeled "A" through "G") that represent a set of virtual auditory display filters that create a virtual auditory display for the wearer. As depicted, the current set of virtual auditory display filters is "VAD C". The wearer may select a different set of virtual auditory display filters by selecting a different icon in group 1164. The wearer may then perform the calibration portion of the calibration and / or personalization process by selecting button 1154 and / or the personalization portion of the calibration and / or personalization process by selecting button 1160.

[0208] The user interface 1150 also allows the wearer to select a custom set of digital filters for the virtual auditory display system 102 to be used to generate a virtual auditory display. The wearer may do this by selecting a button 1168 labeled "Upload," which allows the wearer to upload a file containing the custom set of digital filters to the virtual auditory display system 102. The user interface element 1166 may then display the name of the file. This functionality may be desirable for wearers who already have a custom HRTF and wish to have the virtual auditory display system 102 utilize their custom HRTF.

[0209] Figure 12 is a series of images 1200 illustrating exemplary use cases of the virtual auditory display filter technology described herein in several embodiments. One exemplary use case is for improving virtual surround sound for television or movies using only a pair of speakers, as depicted in image 1202. A group of exemplary use cases relates to producing or listening to music. Image 1210 depicts an exemplary use case of listening to music on headphones. The virtual auditory display filter technology may render music played on headphones as indistinguishable from music played using a physical surround sound system.

[0210] Image 1218 depicts the use of virtual auditory display filter technology in a virtual monitor to provide musicians with noise isolation, sound quality, and virtualization. Image 1204 depicts the use of virtual auditory display filter technology to mix music in any acoustic environment. Image 1220 depicts how virtual auditory display filter technology can provide listeners with a listening experience that revitalizes the music they love. Image 1212 depicts the use of virtual auditory display filter technology in games to provide an immersive gaming experience. Virtual auditory display filter technology can enable users to hear sounds originating from locations not shown on the user's display, and thus improve user perception.

[0211] Another group of exemplary use cases relates to military, non-military (e.g., first responders such as police and fire departments), and / or other organizational applications. For example, military personnel may use military radio systems to communicate with fellow soldiers, commanders, and other military personnel. This technology may be used in scenarios including military operations, emergency services, aviation, maritime, and others.

[0212] Figure 1206 depicts virtual auditory display filter technology that provides improved voice pickup and voice display for organized communications. Figure 1214 depicts virtual auditory display filter technology that provides an extension of visual instrumentation using auditory signals in maritime operations. Figure 1222 depicts virtual auditory display filter technology that provides a hyper-realistic virtual audio environment to facilitate virtual training for military and / or non-military personnel.

[0213] Image 1208 depicts a virtual auditory display filter technology that provides audio enhancements for direction recognition for combat infantry situational awareness. Image 1216 depicts a virtual auditory display filter technology that provides audio enhancements for direction recognition for air force direction control. For example, a pilot may use localization of virtual beacons to assist in situational awareness. Image 1224 depicts a virtual auditory display filter technology that provides audio enhancements for hyper-situational awareness for unmanned aerial vehicle operations.

[0214] Another exemplary use case for virtual auditory display filtering technology involves telephone calls or video conferences. For example, multiple people may be speaking simultaneously in a telephone call or video conference, making it difficult for the listener to focus on the speaker they wish to hear, which can lead to confusion and misunderstanding. This technology allows users to actually select the speaker they wish to hear through simple head movements or other gestures. This attention selection mechanism can help avoid confusion and make meetings more productive.

[0215] As another example, air traffic controllers use radio messaging to communicate with pilots. Air traffic controllers monitor the position, speed, and altitude of aircraft within their assigned airspace visually and by radar, and give instructions to pilots via radio. Often, air traffic controllers need to communicate with multiple pilots simultaneously. Today, these situations requiring communication with multiple pilots are handled by physical switchboards that do not allow for user-directed attention selection. This technology could enable air traffic controllers to localize radio communications so that seamless attention selection exists using gestures (e.g., head or hand movements) or other actions. In a simple example, multiple radio signals are statically placed in unique virtual locations. Air traffic controllers then gaze at these predefined locations and listen to the radio signals. In another example, radio signals are dynamically updated with the aircraft's position, speed, and altitude.

[0216] Other exemplary use cases include virtual auditory display notifications for localizing notifications such as voice, text-to-speech messages, email alerts, phone messages, or other audio notifications; spatial navigation using virtual auditory display cues to communicate the direction and distance of virtual or real objects, which can also be used for wayfinding or orientation recognition; and spatial ambience to provide users with a virtual sound environment that can be mixed with local or virtual sounds (for example, to experience music as if they were in a concert hall). Other exemplary use cases of virtual auditory display filtering technology are possible.

[0217] Figures 13A and 13B illustrate methods for personalizing a digital filter in several embodiments, which involve providing an action (e.g., playing a sound) and detecting user perception of the action. As described herein, a user may indicate perception of an action in various ways, such as by turning their head, making one or more gestures, indicating the location of a sound on a graphical user interface, and so on.

[0218] Figure 13A depicts Figure 1300, showing how the location of a sound in the virtual auditory space may be perceived by the user, as indicated by the azimuth angle 1314 and the elevation angle 1316. Figure 13B depicts Figure 1350, showing how the location of a sound in the virtual auditory space may be perceived by the user, as indicated by the distance 1318 and the elevation angle 1316. In both Figures 1300 and 1350, user 1302 is wearing a first ear-mounted device 102a and a second ear-mounted device 102b (not shown in Figures 13A and 13B). The virtual auditory display system 102 may provide user 1302 with commands to trace the sound in the virtual auditory space with the user's head as the user listens to the sound. The virtual auditory display system 102 may then generate audio signals that cause the first ear-mounted device 102a and the second ear-mounted device 102b to output one or more sounds having location 1304 in the virtual auditory space. As user 1302 hears one or more sounds, user 1302 may turn their head to a perceived location 1306, which may be different from location 1304, where user perceives the sound to be coming from. When turning their head to a perceived location 1306, it may appear as if user 1302 is fixating on the direction in which user 1302 perceives the presence of the sound. Other gestures that user 1302 may make with their head include shaking it up and down, shaking it from side to side, tilting it, and turning it. Other head gestures will become apparent.

[0219] As user 1302 moves their head to face one or more perceived sound locations 1306, one or both of the first ear-mounted device 102a and the second ear-mounted device 102b may detect user 1302's head orientation (for example, using an IMU sensor system 252 and / or a magnetometer 254). The virtual auditory display system 102 may use the detected head orientation to determine the perceived location 1306. The virtual auditory display system 102 may then determine a delta 1308 between location 1304 and the perceived location 1306. The virtual auditory display system 102 may calculate the delta 1308 based on the difference between the azimuth angle 1314 and / or elevation angle 1316 of location 1304 and the azimuth angle 1314 and / or elevation angle 1316 of the perceived location 1306.

[0220] User 1302 may use other gestures to indicate distance, such as extending their arm by a specified amount, and the virtual auditory display system 102 may use such gestures to determine a delta 1308 based on the difference between the distance 1318 of location 1304 and the distance 1318 of perceived location 1306.

[0221] The virtual auditory display system 102 may generate audio signals that cause the first ear-mounted device 102a and the second ear-mounted device 102b to output one or more subsequent sounds whose location in the virtual auditory space changes, as shown by the solid line 1310. The user 1302 may move their head to follow the movement of one or more subsequent sounds, and the perceived location of one or more sounds may change, as shown by the dashed line 1312.

[0222] One or both of the first ear-mounted device 102a and the second ear-mounted device 102b may detect the user 1302's subsequent head orientation as the user moves their head. The virtual auditory display system 102 may use the detected subsequent head orientation to determine the perceived location of one or more subsequent sounds. The virtual auditory display system 102 may then determine the location of one or more subsequent sounds and one or more subsequent deltas between the perceived locations of one or more subsequent sounds.

[0223] The virtual auditory display system 102 stores a delta determined by the virtual auditory display system 102 and may use this delta to modify the digital filter so that the user 1302 perceives that the location of sounds in the virtual auditory space is closer to their actual location in the virtual auditory space. In some embodiments, the virtual auditory display system 102 modifies the digital filter by selecting a different set of digital filters that the virtual auditory display system 102 determines will reduce or minimize the delta for the user 1302. The virtual auditory display system 102 may then use a different set of digital filters for the user 1302.

[0224] In some embodiments, the virtual auditory display system 102 modifies the parameters of the digital filter. For example, the virtual auditory display system 102 may modify parameters such as the center frequency, gain, and / or q. For example, user 1302 may have an elevation delta of several degrees. The virtual auditory display system 102 may modify the elevation of the virtual sound object by modifying the center frequency of a notch, a pair of notches, or a group of notches (see, for example, Figure 6B), thereby reducing the elevation delta. The virtual auditory display system 102 may then utilize the modified digital filter during real-time audio playback.

[0225] The virtual auditory display system 102 may repeat the personalization procedure once or more times until it determines that the delta is within a specific range or threshold.

[0226] In some embodiments, the virtual auditory display system 102 and / or other devices may capture other actions that the user may take in response to hearing sounds in the virtual auditory space, in order to indicate where the user perceives the location of the sound. Examples of such actions may include voice responses by the user, and gestures made by the user using parts of the user's body other than the user's head (e.g., pointing, shaking, clapping, tapping, and hand signals using the user's fingers or arms). Such actions may be captured by devices connected to the virtual auditory display system 102, such as microphones, cameras, or motion detection devices.

[0227] Other exemplary actions include a user indicating the perceived location of a sound using a graphical user interface and / or user input device of a digital device such as a telephone, tablet, laptop, or desktop computer. For example, the virtual auditory display system 102 may provide a graphical user interface that graphically represents a virtual auditory space for the user, and the user may use an input device (mouse, keyboard, touchscreen, and / or voice commands) to indicate the perceived location of a sound on the graphical representation of the virtual auditory space. It will be understood that there are various ways to capture user actions in response to the user's perception of the sound's location, and that the virtual auditory display system 102 may utilize various methods in cooperation with other devices of its choice.

[0228] Figure 14A illustrates a method 1400 for personalizing a digital filter in several embodiments. A virtual auditory display system 102 may perform method 1400. Method 1400 begins in step 1402, where the virtual auditory display system 102 (e.g., binauralizer 138) receives a personalization audio signal having a first location in the virtual auditory space. In step 1404, the virtual auditory display system 102 (e.g., binauralizer 138) determines a first specific first location in the virtual auditory space based on the first location.

[0229] In step 1406, the virtual auditory display system 102 selects one or more specific combined first digital filters from a first set of combined first digital filters and one or more specific combined second digital filters from a first set of combined second digital filters, based on a first specific first location.

[0230] In step 1408, the virtual auditory display system 102 applies a specific combination of first digital filters to the personalization audio signal to obtain a first processed personalization audio signal, and a specific combination of second digital filters to the personalization audio signal to obtain a second processed personalization audio signal.

[0231] In step 1410, the virtual auditory display system 102 generates a first output audio signal for the left ear-worn device based on a first processed personalization audio signal, and a second output audio signal for the right ear-worn device based on a second processed personalization audio signal. In step 1412, the left ear-worn device outputs a first sound based on the first output audio signal, and the right ear-worn device outputs a second sound based on the second output audio signal.

[0232] In step 1414, one or both of the left and right ear-mounted devices detect the head orientation of the user wearing the left and right ear-mounted devices. In step 1416, the virtual auditory display system 102 determines a second specific first location in the virtual auditory space based on the head orientation.

[0233] In step 1418, the virtual auditory display system 102 determines a delta between a first specific first location and a second specific first location. In step 1420, the virtual auditory display system 102 selects a second set of combined first digital filters and a second set of combined second digital filters based on the delta. The virtual auditory display system 102 may use the second set of combined first digital filters and the second set of combined second digital filters while receiving a subsequent input audio signal.

[0234] Figure 14B illustrates a method 1450 for personalizing a digital filter in several embodiments. Method 1450 includes certain steps that may be generally similar to certain steps of Method 1400. A virtual auditory display system 102 (e.g., various components of the virtual auditory display system 102) may perform Method 1450.

[0235] In step 1452, the virtual auditory display system 102 receives a set of multiple first digital filters. In step 1454, the virtual auditory display system 102 receives a set of multiple second digital filters. For each of the multiple virtual auditory space locations, there exists one or more first digital filters and one or more second digital filters.

[0236] In step 1456, the virtual auditory display system 102 receives personalization information about the user. The personalization information may include the user's perception of user instruction actions or acoustic cues, acoustic quality information, user anatomical measurements, user demographic information, and / or user audiometry measurements.

[0237] In step 1458, the virtual auditory display system 102 modifies a set of several first digital filters based on personalization information about the user. In step 1460, the virtual auditory display system 102 modifies a set of multiple second digital filters based on personalization information about the user.

[0238] In some embodiments, modifying a set of multiple first digital filters based on personalization information includes modifying one or more first center frequencies of the multiple first digital filters. Furthermore, modifying a set of multiple second digital filters based on personalization information includes modifying one or more second center frequencies of the multiple second digital filters.

[0239] In some embodiments, modifying a set of multiple first digital filters based on personalization information includes selecting a different set of multiple first digital filters. Furthermore, modifying a set of multiple second digital filters based on personalization information includes selecting a different set of multiple second digital filters.

[0240] The virtual auditory display system 102 may provide a calibration and / or personalization process that enables the wearer of the virtual auditory display device to calibrate the virtual auditory display device and / or personalize the virtual auditory display provided by the virtual auditory display device.

[0241] The calibration and / or personalization process may include a calibration unit and a personalization unit. The virtual auditory display device may include an inertial measurement unit (IMU). Calibrating the virtual auditory display device may mean calibrating the IMU. Personalizing the virtual auditory display may mean selecting a set of virtual auditory display filters for the wearer and / or modifying an existing set of virtual auditory display filters so that the virtual auditory display provided by the virtual auditory display device is customized to the wearer. The virtual auditory display system 102 may allow the wearer to perform both the calibration and personalization units of the calibration and / or personalization process, only the calibration unit, or only the personalization unit.

[0242] Figures 15A to 15C depict exemplary user interfaces 1500 for calibrating a virtual auditory display device in several embodiments. The virtual auditory display device may be a virtual auditory display device 100 including a first ear-mounted device 102a and a second ear-mounted device 102b. Figures 15A and 15F are illustrated with reference to the virtual auditory display device 100, but other virtual auditory display devices may be calibrated and / or personalized.

[0243] A virtual auditory display system 102 (e.g., a user interface module 210) may provide a user interface 1500. The wearer may initiate the calibration and / or personalization process by selecting a button labeled "Start" displayed by the virtual auditory display system 102 (not shown in Figures 15A-15C). Figure 15A depicts a user interface element 1502 indicating a point in the calibration section of the calibration and / or personalization process in which the wearer is placed, and a user interface 1500 providing instructions 1504 for the wearer.

[0244] Figure 15B depicts a user interface 1500 that provides a first circle 1506a and a second circle 1506b. The virtual auditory display system 102 may move up and down on the user interface 1500 over the first circle 1506a and / or the second circle 1506b and command the wearer to move their head up and down to follow the first circle 1506a and the second circle 1506b.

[0245] Figure 15C depicts a user interface 1500 that provides a circle 1508. The virtual auditory display system 102 may instruct the wearer to move the circle 1508 up and down on the user interface 1500 and to move their head up and down to follow the circle 1508. In addition or alternatively, the virtual auditory display system 102 may instruct the wearer to move the circle 1508 left and right on the user interface 1500 and to move their head left and right to follow the circle 1508.

[0246] While the virtual auditory display system 102 performs the calibration unit of the calibration and / or personalization process, the virtual auditory display system 102 may receive a detection of the wearer's head orientation from the virtual auditory display device 100 based on data acquired from the IMU-based sensor system and / or other sensors of the first ear-mounted device 102a and / or the second ear-mounted device 102b. The virtual auditory display system 102 may use the detection of head orientation and other factors such as a known or estimated distance from the display providing the user interface 1500, the width and height of the display, the positions of the first circle 1506a, the second circle 1506b, and / or circle 1508, and / or other data from the IMU-based sensor system for calibrating the IMU-based sensor system.

[0247] Figures 15D to 15F depict exemplary user interfaces 1550 for personalizing a virtual auditory display provided by a virtual auditory display device in several embodiments. A virtual auditory display system 102 (e.g., a user interface module 210) may provide the user interface 1550.

[0248] The wearer of the virtual auditory display device 100 may, after completing the calibration section, start the personalization section of the calibration and / or personalization process. The virtual auditory display system 102 may have the virtual auditory display device 100 play sounds in several locations (e.g., five locations). The sounds may include sounds produced by objects that appear to the wearer to be moving around their head, such as airplanes, helicopters, birds, and other flying creatures. Figure 15D depicts a user interface 1550 that provides the wearer with a command 1554 instructing them to turn their nose towards the source of each sound as the virtual auditory display device 100 plays sounds. The wearer may start the personalization section of the calibration and / or personalization process by selecting a button 1556 labeled "Continue".

[0249] Figure 15E depicts a user interface 1550 that provides a user interface element 1552 indicating a point in the personalization section of the calibration and / or personalization process in which the wearer is located, and a command 1554. Figure 15F depicts a user interface 1550 having a user interface element 1552 indicating that the wearer has located a sound played by the virtual auditory display device 100 at a first location. The virtual auditory display system 102 may cause the virtual auditory display device 100 to play a sound at a subsequent location and update the user interface 1550 accordingly.

[0250] While the virtual auditory display system 102 performs the personalization portion of the calibration and / or personalization process, the virtual auditory display system 102 may receive detection of the wearer's head orientation from the virtual auditory display device 100 based on data acquired from the IMU-based sensor system and / or other sensors of the first ear-mounted device 102a and / or the second ear-mounted device 102b. The virtual auditory display system 102 may calculate one or more deltas using the detection of head orientation and the location of the sound generated by the virtual auditory display system 102, as described with reference to, for example, Figures 13A and 13B. The virtual auditory display system 102 may use the calculated one or more deltas to select a set of virtual auditory display filters and estimate the spatialization accuracy of the virtual auditory display for the wearer.

[0251] Figures 15G to 15J depict exemplary user interfaces 1570 for providing information regarding the calibration of a virtual auditory display device and the personalization of the virtual auditory display of the virtual auditory display device in several embodiments. User interface 1570 includes a recommendation 1572 of a set of virtual auditory display filters. In some embodiments, as described herein with reference to, for example, Figures 13A and 13B, the virtual auditory display system 102 may select a set of virtual auditory display filters from a plurality of sets of virtual auditory display filters based on the results of the calibration and / or personalization process. User interface 1570 also includes an estimate 1574 of the spatialization accuracy of the virtual auditory display for the wearer.

[0252] The virtual auditory display system 102 may categorize the spatial accuracy of the virtual auditory display for the wearer based on estimates 1574, such as “very good” (Figure 15G), “medium” (Figure 15H), and “poor” (Figure 15I). The virtual auditory display system 102 may provide recommendations to rerun the calibration and / or personalization sections of the calibration and / or personalization process and / or use custom filters. The user interface 1570 also includes a button 1576 labeled “Continue” that the wearer can select to return to the user interface 1100 depicted in Figures 11A and 11B.

[0253] Although the virtual auditory display system 102 is described as using circles, the virtual auditory display system 102 may utilize other visual user interface elements in the calibration and / or personalization process. Furthermore, although the virtual auditory display system 102 is described as receiving head orientation detection from the virtual auditory display device 100 in the calibration and / or personalization process, the virtual auditory display system 102 may receive head orientation detection from other devices connected to the virtual auditory display system 102, such as cameras, motion detection devices, virtual reality headsets, and the like.

[0254] One advantage of the calibration and / or personalization process is that the virtual auditory display system 102 can personalize a set of virtual auditory display filters for a wide range of individuals. The virtual auditory display system 102 may personalize a set of virtual auditory display filters by modifying the set of virtual auditory display filters. The virtual auditory display system 102 may have multiple sets of virtual auditory display filters pre-configured and may modify a set of virtual auditory display filters by selecting a different set of virtual auditory display filters based on the results of a calibration and / or personalization process for a particular user.

[0255] In addition or alternatively, the virtual auditory display system 102 may modify the set of virtual auditory display filters by modifying the digital filters or functions included in the set of virtual auditory display filters. For example, if the set of virtual auditory display filters includes digital filters, the virtual auditory display system 102 may modify the parameters of the digital filters, such as center frequency, gain, q, algorithm type, or other parameters, based on the results of a calibration and / or personalization process for a particular user.

[0256] Personalization of virtual auditory display filters enables a wide range of individuals to experience immersive, accurately rendered sounds in the virtual auditory space. Furthermore, such individuals would not need to possess HRTFs generated for them using potentially difficult and / or unreliable physical measurement procedures. Such individuals could simply obtain a personalized set of virtual auditory display filters by having the virtual auditory display system 102 perform a calibration and / or personalization process for them. Modifications to the virtual auditory display filters can be performed during the initial setup procedure for that individual and at any subsequent point in the individual's use of the virtual auditory display system 102 and / or virtual auditory display device 100.

[0257] One advantage of virtual auditory display filters is that, compared to existing technologies, sounds can be rendered in a much larger number of locations within the virtual auditory space. For example, a 9.1.6 configuration may have 16 virtual speakers and therefore be limited to precisely rendering sounds only in those 16 virtual speaker locations. Such a configuration may render sounds from other locations by smearing sounds from those virtual speaker locations to represent them, but such artifacts may be noticeable to the listener.

[0258] In contrast, a virtual auditory display filter may be capable of rendering sound at a far greater number of locations. For example, using locations in 1-degree increments of azimuth and elevation yields 65,160 locations. However, the technique described may generate virtual auditory display filters at smaller increments, resulting in even more locations where the virtual auditory display filter can render sound. Moreover, a typical method renders sound at a modeled distance of 1 meter from a center point representing the listener. The technique described can generate virtual auditory display filters for any number of distances from the center point. Therefore, the technique described can accurately render sound at varying distances.

[0259] One advantage of the described technology is that it accurately renders virtual auditory display sounds in a virtual auditory space, meaning that the sound is perceived by the listener as coming from the location intended by the sound creator. Another advantage of the described technology is that it can be used with any on-ear device such as headphones, headsets, and earbuds. Another advantage is that the virtual auditory display sounds are of high quality and clarity. Another advantage is that the described technology can emphasize or deem sounds in specific areas or locations in the virtual auditory space to focus the listener's attention on those specific areas or locations. Such techniques can enhance the listener's hearing ability and allow the listener to hear sounds that they would not otherwise hear.

[0260] Another advantage of the technology described is that any digital device with suitable storage and processing power can store and apply virtual auditory display filters. As described herein, a general-purpose computing device such as a laptop or desktop computer may store virtual auditory display filters and apply them to an audio signal to generate a processed audio signal. The laptop or desktop computer may then transmit the processed audio signal to an ear-worn device to generate sound based on the processed audio signal. Similarly, a digital device such as a telephone, tablet, or virtual reality headset may store virtual auditory display filters, apply them to an audio signal to generate a processed audio signal, and transmit the processed audio signal to an ear-worn device.

[0261] In addition or alternatively, an ear-worn device such as the virtual auditory display device 100 described herein may store and apply a virtual auditory display filter. The ear-worn device may receive an input audio signal from a digital device to which it is paired, such as a telephone or tablet, or from a cloud-based service. The ear-worn device may apply the stored virtual auditory display filter to the input audio signal to generate a processed audio signal and output a virtual auditory display sound based on the processed audio signal. In another example, the cloud-based service may store and apply the virtual auditory display filter to generate a processed audio signal and transmit the processed audio signal to the ear-worn device. Other advantages will become apparent.

[0262] Figure 16 depicts a block diagram of an illustrative digital device 1600 according to several embodiments. The digital device 1600 is shown in the form of a general-purpose computing device. The digital device 1600 includes at least one processor 1602, RAM 1604, a communication interface 1606, an input / output device 1608, storage 1610, and a system bus 1612 that connects various system components, including storage 1610, to at least one processor 1602. A system such as a computing system may be one or more digital devices 1600, or include them.

[0263] System Bus 1612 represents any one or more of several types of bus structures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. Examples, but not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0264] The digital device 1600 typically includes a variety of computer system-readable media, such as computer system-readable storage media. Such media may be any available media accessible by any of the systems described herein, and may include both volatile and non-volatile media, and removable and non-removable media.

[0265] In some embodiments, at least one processor 1602 is configured to execute executable instructions (e.g., programs). In some embodiments, at least one processor 1602 includes circuitry or any processor capable of processing executable instructions.

[0266] In some embodiments, RAM 1604 stores programs and / or data. In various embodiments, working data is stored in RAM 1604. The data in RAM 1604 may be cleared or eventually transferred to storage 1610, for example, before resetting and / or turning off the power to the digital device 1600.

[0267] In some embodiments, the digital device 1600 is connected to a network via a communication interface 1606. The digital device 1600 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet).

[0268] In some embodiments, the input / output device 1608 is any device that inputs data (e.g., mouse, keyboard, stylus, sensor, etc.) or outputs data (e.g., speaker, display, virtual reality headset).

[0269] In some embodiments, the storage 1610 may include computer system-readable media in the form of non-volatile memory such as read-only memory (ROM), programmable read-only memory (PROM), solid-state drive (SSD), flash memory, and / or cache memory. The storage 1610 may further include other removable / non-removable, volatile / non-volatile computer system storage media. For illustrative purposes only, the storage 1610 may be provided for reading and writing to a non-removable, non-volatile magnetic medium. The storage 1610 may include non-temporary computer-readable media, or a plurality of non-temporary computer-readable media, which store programs or applications for performing functions such as those described herein with reference to Figures 2A, 2B, and 3B, for example. Although not shown, magnetic disk drives for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media can be provided. In such cases, each can be connected to the system bus 1612 by one or more data media interfaces. As will be further described below, the storage 1610 may include at least one program product having a set of program modules (e.g., at least one) configured to perform the functions of embodiments of the present invention. In some embodiments, RAM 1604 is found within the storage 1610.

[0270] A program / utility having a set (at least one) of program modules such as the virtual auditory display system 102 may be stored in the storage 1610, by way of example and not limitation, together with an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules generally execute the functions and / or methodologies of embodiments of the present invention as described herein.

[0271] Although not shown, it should be understood that other hardware and / or software components may be used in combination with the digital device 1600. By way of example, but not limited thereto, microcode, device drivers, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archive storage systems, etc. may be mentioned.

[0272] Exemplary embodiments are described herein in detail with reference to the accompanying drawings. However, the present disclosure can be implemented in various ways and should not be construed as limited to the embodiments disclosed herein. On the contrary, those embodiments are provided for a full and complete understanding of the present disclosure and to fully convey the scope of the present disclosure.

[0273] It will be understood that aspects of one or more embodiments may be embodied as a system, method, or computer program product. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that may all be referred to herein generically as a "circuit," "module," or "system." Further, aspects may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0274] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium, for example, but not limited to, may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a solid state drive (SSD), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can store or retain a program or data for use by or in connection with an instruction execution system, apparatus, or device.

[0275] A temporary computer-readable signal medium may include, for example, a propagated data signal in which a computer-readable program code is embodied, either in the baseband or as part of a carrier wave. Such propagated signals can take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof.

[0276] Program code, as embodied on a computer-readable medium, may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic cable, RF, or any suitable combination thereof.

[0277] Computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, C++, Python®, or similar, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer program code may be fully executed on any of the systems described herein, or any combination of the systems described herein.

[0278] Aspects of the present invention will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to create a machine, thereby creating means for implementing functions / operations specified in one or more blocks of a flowchart and / or block diagram when the instructions are executed via the processor of the computer or other programmable data processing device.

[0279] Furthermore, these computer program instructions may be stored in a computer-readable medium that can instruct a computer, other programmable data processing device, or other device to function in a particular manner, thereby producing a product containing instructions that implement a specified mode of function / operation in one or more blocks of a flowchart and / or block diagram.

[0280] Furthermore, computer program instructions may be loaded into a computer, another programmable data processing device, or another device to execute a series of operational steps on that computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on that computer or other programmable device provide a process for implementing functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0281] While specific examples are provided above for illustrative purposes, various equal forms of modification are possible. For example, while processes or blocks are presented in a given order, alternative implementations may employ systems with steps, routines, or blocks in a different order, and some processes or blocks may be removed, moved, added, subdivided, combined, and / or modified to provide alternatives or partial combinations. Each of these processes or blocks may be implemented in a variety of different ways. Furthermore, while processes or blocks are sometimes shown as being executed sequentially, these processes or blocks may instead be executed or implemented simultaneously or in parallel, or executed at different points in time. Moreover, any specific numbers described herein are merely examples: alternative implementations may employ different values ​​or ranges.

[0282] Throughout this specification, multiple examples may implement a component, operation, or structure described as a single example. Structures and functionalities presented as separate components in illustrative configurations may be implemented as combined structures or components. Similarly, structures and functionalities presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements are included within the scope of the subject matter of this specification. Furthermore, any particular numbers described herein are merely examples: alternative implementations may employ different values ​​or ranges.

[0283] Components may be described or illustrated as being housed within or connected to other components. Such descriptions or illustrations are merely examples, and other configurations may achieve the same or similar functionality. Components may be described or illustrated as being “combined,” “combinable,” “operatably combined,” “communicatively combined,” and similar to other components. Such descriptions or illustrations should be understood as indicating that such components may cooperate or interact with one another and may have direct or indirect physical, electrical, or communication contact with one another.

[0284] Components may be described or illustrated as "configured to," "adapted to," "operable to," "configurable to," "adaptable to," "operable to," and similar. Such descriptions or illustrations should be understood to encompass the component in both the active and disabled or standby states, unless otherwise required by context.

[0285] The use of “or” in this disclosure is not intended to be understood as an exclusive “or.” Rather, “or” should be understood as including “and / or.” For example, the expression “to provide products or services” is intended to be understood as having several meanings: “to provide products,” “to provide services,” and “to provide products and services.”

[0286] Various modifications may be made, and it may become apparent that other embodiments can be used without departing from the broader scope of the discussion herein. For example, the virtual auditory display system 102 may utilize a group of FIR filters for each of specific locations in the virtual auditory space and a group of IIR filters for each of other specific locations in the virtual auditory space. As another example, the virtual auditory display system 102 may provide the audio signal to any device capable of directing sound to the listener's ears. As yet another example, the virtual auditory display device may be any device or set of devices (such as a pair of speakers) capable of producing sound based on the output audio signal generated by the virtual auditory display system 102.

[0287] Therefore, these and other variations of the exemplary embodiments are intended to be covered by this disclosure.

Claims

1. A computer program, when executed by one or more processors of a system, comprises executable instructions that cause the system to perform a method, wherein the method is: For each of a plurality of virtual auditory space locations, a step of generating one or more first digital filters, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies based on a first substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more first notch filters, when applied to a first audio signal, are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies; For each of the plurality of virtual auditory space locations, the step of generating one or more second digital filters, wherein the one or more second digital filters include one or more second notch filters, the one or more second notch filters include one or more second center frequencies, the one or more second center frequencies are based on a second substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more second notch filters, when applied to a second audio signal, are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies; In the step of receiving an audio signal, the audio signal has one or more audio sub-signals, and the audio sub-signals are associated with a virtual auditory space location; For each of the one or more audio sub-signals mentioned above: A step of selecting a specific one or more first digital filter and a specific one or more second digital filters based on the virtual auditory space location associated with the audio sub-signal; A step of applying the aforementioned specific one or more first digital filters to the audio sub-signal to obtain a first processed audio sub-signal; and A step of applying one or more of the aforementioned specific second digital filters to the audio sub-signal to obtain a second processed audio sub-signal; A step of generating a first output audio signal for a first device based on a plurality of first processed audio sub-signals; A step of generating a second output audio signal for a second device based on a plurality of second processed audio sub-signals; and The step of providing the first output audio signal to the first device and the second output audio signal to the second device. A computer program that has [a certain characteristic].

2. The aforementioned virtual auditory space location is a first virtual auditory space location, and the method is: The step of receiving the user's head orientation; and For each of the one or more audio sub-signals, a second virtual auditory space location is determined based on the first virtual auditory space location and the head orientation associated with the audio sub-signal. It further possesses, The computer program according to claim 1, wherein the step of selecting one or more specific first digital filters and one or more specific second digital filters based on the virtual auditory space location associated with the audio sub-signal includes the step of selecting one or more specific first digital filters and one or more specific second digital filters based on the second virtual auditory space location.

3. The specified one or more first digital filters are the first specified one or more first digital filters, the specified one or more second digital filters are the first specified one or more second digital filters, the user's head orientation is the user's first head orientation, and the method is: The step of receiving a personalization audio signal associated with a third virtual auditory space location; A step of selecting a second specific one or more first digital filters and a second specific one or more second digital filters based on the third virtual auditory space location; A step of applying one or more of the second specific first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; A step of applying one or more of the second specific second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; A step of generating a third output audio signal for the first device based on the first processed personalization audio signal; A step of generating a fourth output audio signal for the second device based on the second processed personalization audio signal; The step of providing the third output audio signal to the first device and the fourth output audio signal to the second device; The step of receiving the second head orientation of the user; A step of determining a fourth virtual auditory spatial location based on the second head orientation; A step of determining the delta between the third virtual auditory space location and the fourth virtual auditory space location; and A step of modifying the one or more first digital filters and the one or more second digital filters based on the delta. The computer program according to claim 2, further comprising the above.

4. The computer program according to claim 3, wherein the step of modifying the one or more first digital filters and the one or more second digital filters based on the delta includes the step of modifying the one or more first center frequencies on which the one or more first notch filters are based and the one or more second center frequencies on which the one or more second notch filters are based.

5. The method further comprises the step of generating a first notch mask and a second notch mask using one or more image processing algorithms, wherein the first notch mask specifies a first gain modifier as a function of the virtual auditory space location, and the second notch mask specifies a second gain modifier as a function of the virtual auditory space location: The one or more first notch filters include the one or more first center frequencies and a first gain such that it is modified by the first gain modifier, and the one or more first notch filters, when applied to the first audio signal, are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain. The computer program according to claim 1, wherein the one or more second notch filters include the one or more second center frequencies and a second gain such that it is modified by the second gain modifier, and the one or more second notch filters are configured, when applied to the second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain.

6. The computer program according to claim 5, wherein the one or more image processing algorithms include one or more of a Gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and a segmentation function.

7. The aforementioned method is: The stage of receiving the selection of the acoustic environment; and A step in which a first acoustic environment digital filter and a second acoustic environment digital filter are determined based on the aforementioned acoustic environment. It further possesses, The computer program according to claim 1, wherein for each of the one or more audio sub-signals, the step of applying the specific one or more first digital filters to the audio sub-signal to obtain the first processed audio sub-signal includes the step of applying the specific one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal, and the step of applying the specific one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal includes the step of applying the specific one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

8. The computer program according to claim 7, wherein the acoustic environment is represented by one or more ambisonic arrays, and the step of determining the first acoustic environment digital filter and the second acoustic environment digital filter based on the acoustic environment includes the step of determining the first acoustic environment digital filter and the second acoustic environment digital filter based on the one or more ambisonic arrays.

9. The computer program according to any one of claims 1 to 8, wherein the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

10. The computer program according to any one of claims 1 to 8, wherein the first device includes a first ear-mounted device, and the second device includes a second ear-mounted device.

11. A system comprising at least one processor and at least one memory, wherein the at least one memory, when executed by the at least one processor, provides to the system: For each of a plurality of virtual auditory space locations, one or more first digital filters are generated, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies being based on a first substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more first notch filters being configured, when applied to a first audio signal, to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies; For each of the plurality of virtual auditory space locations, one or more second digital filters are generated, the one or more second digital filters comprising one or more second notch filters, the one or more second notch filters comprising one or more second center frequencies, the one or more second center frequencies being based on a second substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more second notch filters being configured, when applied to a second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies; Receiving an audio signal, the audio signal having one or more audio sub-signals, the audio sub-signals being associated with a virtual auditory space location; For each of the one or more audio sub-signals mentioned above: Selecting one or more specific first digital filters and one or more specific second digital filters based on the virtual auditory space location associated with the audio sub-signal; Applying the aforementioned one or more specific first digital filters to the audio sub-signal to obtain a first processed audio sub-signal; and Applying the aforementioned one or more second digital filters to the audio sub-signal to obtain a second processed audio sub-signal; To generate a first output audio signal for a first device based on a plurality of first processed audio sub-signals; Generating a second output audio signal for a second device based on a plurality of second processed audio sub-signals; and To provide the first output audio signal to the first device and the second output audio signal to the second device. A system having executable instructions that cause it to perform an action.

12. The virtual auditory space location is a first virtual auditory space location, and the executable instruction, when executed by the at least one processor, is transmitted to the system: Receiving the user's head orientation; and For each of the one or more audio sub-signals, a second virtual auditory space location is determined based on the first virtual auditory space location and the head orientation associated with the audio sub-signal. Let them do this further, The system according to claim 11, wherein selecting one or more specific first digital filters based on the virtual auditory space location associated with the audio sub-signal includes selecting one or more specific first digital filters based on a second virtual auditory space location, and selecting one or more specific second digital filters based on the virtual auditory space location associated with the audio sub-signal includes selecting one or more specific second digital filters based on a second virtual auditory space location.

13. The one or more first digital filters are first one or more first digital filters, the one or more second digital filters are first one or more second digital filters, the specific one or more first digital filters are first specific one or more first digital filters, the specific one or more second digital filters are first specific one or more second digital filters, the head orientation is first head orientation, the audio signal having one or more audio sub-signals is first audio signal having first one or more audio sub-signals, and the executable instruction, when executed by the at least one processor, is transmitted to the system: Receiving a personalization audio signal having a third virtual auditory space location; Selecting a second specific one or more first digital filters and a second specific one or more second digital filters based on the third virtual auditory space location; Applying the aforementioned second specific one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; Applying one or more of the aforementioned second specific second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; To generate a third output audio signal for the first device based on the first processed personalization audio signal; To generate a fourth output audio signal for the second device based on the second processed personalization audio signal; To provide the third output audio signal to the first device and the fourth output audio signal to the second device; To receive the second head orientation of the user; Based on the second head orientation, a fourth virtual auditory spatial location is determined; Determining the delta between the third virtual auditory space location and the fourth virtual auditory space location; and Based on the delta, select one or more second first digital filters and one or more second second digital filters, the one or more second first digital filters and the one or more second second digital filters are for use while receiving a second input audio signal having one or more second audio sub-signals. The system according to claim 12, further comprising the following:

14. When the executable instruction is executed by the at least one processor, it causes the system to use one or more image processing algorithms to generate a first notch mask and a second notch mask, the first notch mask specifying a first gain modifier based on the virtual auditory space location, and the second notch mask specifying a second gain modifier based on the virtual auditory space location: The one or more first notch filters are generated using the one or more first center frequencies based on the first substantially sigmoid distribution of center frequencies as a function of virtual auditory spatial locations, and a first gain such that it is modified by the first gain modifier, and the one or more first notch filters, when applied to the first audio signal, are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain. The system according to claim 11, wherein the one or more second notch filters are generated using the one or more second center frequencies based on the second substantially sigmoid distribution of center frequencies as a function of virtual auditory spatial locations, and a second gain such that it is modified by the second gain modifier, and the one or more second notch filters are configured, when applied to the second audio signal, to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain.

15. When the executable instruction is executed by the at least one processor, it is sent to the system: Receiving the selection of the acoustic environment; and Based on the aforementioned acoustic environment, a first acoustic environment digital filter and a second acoustic environment digital filter are determined. Let them do this further, The system according to claim 11, wherein for each of the one or more audio sub-signals, obtaining the first processed audio sub-signal by applying the specific one or more first digital filters to the audio sub-signal includes applying the specific one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal, and obtaining the second processed audio sub-signal by applying the specific one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal includes applying the specific one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

16. The system according to any one of claims 11 to 15, wherein the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

17. The system according to any one of claims 11 to 15, wherein the first device includes a first ear-mounted device, and the second device includes a second ear-mounted device.

18. Method: Steps include generating a first virtual auditory display filter, the first virtual auditory display filter comprising a first set of first functions, one or more of which, when applied to a first audio signal having a first location in a virtual auditory space, generate a first processed audio signal having a first frequency response having one or more first notches at one or more first center frequencies based on the first location, the one or more first notches having one or more first peak-to-trough depths of up to -10 dB; The step of generating a second virtual auditory display filter, the second virtual auditory display filter comprising a second set of second functions, one or more of which, when applied to the first audio signal, generate a second processed audio signal having a second frequency response having one or more second notches at one or more second center frequencies based on the first location, the one or more second notches having one or more second peak-to-trough depths of up to -10 dB; The step of receiving a second audio signal having a second location in the virtual auditory space; Steps include applying the first virtual auditory display filter, which includes a first subset of the first function selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response; Steps include applying the second virtual auditory display filter, which includes a second subset of the second function selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response; The step of providing the third processed audio signal to the first sound output device; and Step 4: Providing the processed audio signal to the second sound output device. A method that includes [a certain feature].

19. The method according to claim 18, wherein the one or more first center frequencies are based on a first substantially sigmoid distribution of center frequencies as a function of location in the virtual auditory space, and the one or more second center frequencies are based on a second substantially sigmoid distribution of center frequencies as a function of location in the virtual auditory space.

20. Further comprising a stage for receiving the user's head orientation: The step of applying the first virtual auditory display filter, which includes the first subset of the first function selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response, includes the step of applying the first virtual auditory display filter, which includes the third subset of the first function selected based on the second location and the head orientation, to the second audio signal to generate the third processed audio signal having a third frequency response, The step of applying the second virtual auditory display filter, which includes the second subset of the second function selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response, comprises the step of applying the second virtual auditory display filter, which includes the fourth subset of the second function selected based on the second location and the head orientation, to the second audio signal to generate the fourth processed audio signal having a fourth frequency response. The method according to claim 18.

21. A step of generating a first notch mask and a second notch mask using one or more image processing algorithms, wherein the first notch mask specifies a first depth modifier as a function of the location in the virtual auditory space, and the second notch mask specifies a second depth modifier as a function of the location in the virtual auditory space; A step of modifying the one or more first peak-to-trough depths based on the first depth modifier; and A step of modifying the one or more second peak-to-trough depths based on the second depth modifier. The method according to claim 18, further comprising:

22. The method according to claim 21, wherein the one or more image processing algorithms include one or more of a Gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and a segmentation function.

23. The method according to claim 21 or 22, wherein the first set of the first function comprises a first infinite impulse response digital filter, and the second set of the second function comprises a second infinite impulse response digital filter.

24. Method: The steps include receiving a set of multiple first digital filters, one or more first digital filters being generated for each of a plurality of virtual auditory space locations, the one or more first digital filters comprising one or more first notch filters, the one or more first notch filters comprising one or more first center frequencies, the one or more first center frequencies being based on a first substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more first notch filters being configured, when applied to a first audio signal, to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies; The step of receiving a set of multiple second digital filters, one or more second digital filters are generated for each of a plurality of virtual auditory space locations, the one or more second digital filters include one or more second notch filters, the one or more second notch filters include one or more second center frequencies, the one or more second center frequencies are based on a second substantially sigmoid distribution of center frequencies as a function of the virtual auditory space locations, and the one or more second notch filters, when applied to a second audio signal, are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies; The step of receiving a personalization audio signal with a virtual auditory space location; A step of selecting one or more specific first digital filters and one or more specific second digital filters based on the virtual auditory space location; A step of applying the aforementioned one or more specific first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; A step of applying the aforementioned specific one or more second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; A step of providing a first output audio signal based on the first processed personalization audio signal to a first device, and a second output audio signal based on the second processed personalization audio signal to a second device; The steps include receiving user perception of a first sound output from the first device and a second sound output from the second device; and A step of modifying the set of multiple first digital filters and the set of multiple second digital filters based on the user perception. A method that includes [a certain feature].

25. The virtual auditory space location is a first virtual auditory space location, and the step of modifying the set of multiple first digital filters and the set of multiple second digital filters based on the user perception is: A step in which a second virtual auditory space location is determined based on the user perception; A step of determining the delta between the first virtual auditory space location and the second virtual auditory space location; and A step of modifying the set of multiple first digital filters and the set of multiple second digital filters based on the delta. The method according to claim 24, having the following characteristics.

26. The method according to claim 25, wherein the step of receiving user perception includes a step of receiving the user's head orientation, and the step of determining the second virtual auditory space location based on the user perception includes a step of determining the second virtual auditory space location based on the user's head orientation.

27. The method according to claim 25, wherein the step of receiving user perception includes the step of receiving one or more gestures of the user, and the step of determining the second virtual auditory space location based on the user perception includes the step of determining the second virtual auditory space location based on one or more gestures of the user.

28. The method according to any one of claims 24 to 27, wherein the set of a plurality of first digital filters is a first set of a plurality of first digital filters, the set of a plurality of second digital filters is a first set of a plurality of second digital filters, the step of modifying the set of a plurality of first digital filters based on user perception includes the step of selecting a second set of a plurality of first digital filters based on user perception, and the step of modifying the set of a plurality of second digital filters based on user perception includes the step of selecting a second set of a plurality of second digital filters based on user perception.

29. The method according to any one of claims 24 to 27, wherein the step of modifying the set of a plurality of first digital filters based on the user perception includes the step of modifying the one or more first center frequencies, and the step of modifying the set of a plurality of second digital filters based on the user perception includes the step of modifying the one or more second center frequencies.

30. The method according to any one of claims 24 to 27, further comprising the steps of: determining a spatial accuracy estimate based on the user perception; and providing the spatial accuracy estimate.

Images