Personalization of head-related transfer function templates for audio content presentation

Abstract

A system for generating a personalized HRTF customized for a user of a head mounted device. The system includes a server and an audio system. The server determines the personalized HRTF based in part on acoustic feature data of the user (e.g., image data, anthropometric features, etc.) and a template HRTF. The server provides the personalized HRTF to the audio system. The audio system uses the personalized HRTF to present spatialized audio content to the user.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 387,897, filed April 18, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0003] background

[0004] The present disclosure relates generally to binaural audio synthesis, and in particular to personalizing head-related transfer functions (HRTFs) to render audio content.

[0005] The sound received at the two ears from a given sound source may be different, depending on the direction and position of the sound source relative to each ear and the room environment in which the sound is perceived. HRTFs characterize the sound received at a person's ears for a specific location (and frequency) of the sound source. Multiple HRTFs are used to characterize how a user perceives sound. In some cases, multiple HRTFs form a high-dimensional data set that relies on tens of thousands of parameters to provide a listener with a perception of the direction of a sound source.

[0006] Overview

[0007] A system for generating a personalized HRTF customized for a user of an audio system (e.g., which can be implemented as part of a head-mounted device (headset)). The system includes a server and an audio system. The server determines the personalized HRTF based in part on the user's acoustic feature data (e.g., image data, anthropometric features, etc.) and a template HRTF. The template HRTF is an HRTF that can be customized (e.g., adding one or more notches) so that it can be personalized for different users. The server provides the personalized HRTF to the audio system. The audio system uses the personalized HRTF to present spatialized audio content to the user. The method described herein may also be embodied as instructions stored on a computer-readable medium.

[0008] According to a first aspect of the present invention, a method is provided, comprising: determining one or more personalized filters based at least in part on acoustic feature data of a user; generating one or more personalized head-related transfer functions HRTFs for the user based on a template HRTF and the determined one or more personalized filters; and providing the generated one or more personalized HRTFs to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

[0009] The method may be executed by a server.

[0010] One or more personalized filters may be determined through machine learning.

[0011] One or more personalized filters may be used to personalize the template HRTF so that it is customized for the user, thereby forming a personalized HRTF.

[0012] Personalization may mean adding one or more notches.

[0013] Determining one or more personalized filters may include using a trained machine learning model and acoustic feature data of the user to determine parameter values ​​of the one or more personalized filters.

[0014] The parameter values ​​of the one or more personalized filters may describe one or more personalized notches in the one or more personalized HRTFs.

[0015] The parameter value may include one or more of: a frequency position, a width in a frequency band centered at the frequency position, and an amount of attenuation caused in the frequency band centered at the frequency position. The parameter value may include each of the list.

[0016] The machine learning model may be trained with image data, anthropometric features, and / or acoustic data, including measurements of HRTFs obtained for a population of users.

[0017] Generating one or more personalized HRTFs for the user based on the template HRTF and the determined one or more personalized filters may include adding at least one notch to the template HRTF using at least one of the one or more personalized filters to generate a personalized HRTF in the one or more personalized HRTFs.

[0018] The template HRTF may be based on a generic HRTF describing a user group, the generic HRTF comprising at least one notch in the frequency range.

[0019] The template HRTF may be generated from the generic HRTF by removing at least one notch so that the template HRTF is a smooth and continuous function over the frequency range.

[0020] The frequency range can be 5kHz to 10kHz.

[0021] There may be at least one notch in the template HRTF outside the frequency range.

[0022] The audio system may be part of the head mounted device.

[0023] Alternatively, the audio system may be separate from and external to the headset.

[0024] According to a second aspect of the present invention, a non-transitory computer-readable medium is provided, which is configured to store program code instructions. When executed by a processor, the program code instructions cause the processor to perform steps including the following operations: determining one or more personalized filters based at least in part on the acoustic feature data of the user; generating one or more personalized head-related transfer functions HRTFs for the user based on a template HRTF and the determined one or more personalized filters; and providing the generated one or more personalized HRTFs to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

[0025] Determining one or more personalized filters may include using a trained machine learning model and acoustic feature data of the user to determine parameter values ​​of the one or more personalized filters.

[0026] The parameter values ​​of the one or more personalized filters may describe one or more personalized notches in the one or more personalized HRTFs.

[0027] The parameter value may include one or more of: a frequency position, a width in a frequency band centered at the frequency position, and an amount of attenuation caused in the frequency band centered at the frequency position. The parameter value may include each of the list.

[0028] The machine learning model may be trained with image data, anthropometric features, and / or acoustic data, including measurements of HRTFs obtained for a population of users.

[0029] Generating one or more personalized HRTFs for the user based on the template HRTF and the determined one or more personalized filters may include adding at least one notch to the template HRTF using at least one of the one or more personalized filters to generate a personalized HRTF in the one or more personalized HRTFs.

[0030] According to a third aspect of the present invention, a method is provided, comprising: receiving, at a head-mounted device, one or more personalized head-related transfer functions (HRTFs) of a user of the head-mounted device; retrieving audio data associated with a target sound source direction relative to the head-mounted device; applying the one or more personalized HRTFs to the audio data to render the audio data as audio content; and presenting the audio content by a speaker assembly of the head-mounted device, wherein the presented audio content is spatialized so that it sounds as if it originates from the target sound source direction.

[0031] One or more HRTFs may be received from a server. The head mounted device may retrieve the audio data. The head mounted device may apply the one or more personalized HRTFs to the audio data.

[0032] The method may further include: capturing acoustic feature data of the user; and transmitting the captured acoustic feature data to a server, wherein the server uses the captured acoustic feature data to determine one or more personalized HRTFs, and the server provides the one or more personalized HRTFs to the head mounted device.

[0033] A system for generating a personalized HRTF customized for a user of an audio system (e.g., which may be implemented as part of a head-mounted device) is also described. The system may include a server and an audio system. The server may determine the personalized HRTF based in part on acoustic feature data of the user (e.g., image data, anthropometric features, etc.) and a template HRTF. The template HRTF may be an HRTF that can be customized (e.g., adding one or more notches) so that it can be personalized for different users. The server may provide the personalized HRTF to the audio system. The audio system may use the personalized HRTF to present spatialized audio content to the user.

[0034] The methods described herein may also be embodied as instructions stored on a computer-readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a perspective view of the elevation of a sound source from a user's viewpoint according to one or more embodiments.

[0037] Figure 2 Shown are example depictions of three HRTFs parameterized by a user's sound source elevation angle in accordance with one or more embodiments.

[0038] Figure 3 is a schematic diagram of a high-level system environment for generating personalized HRTFs according to one or more embodiments.

[0039] Figure 4 is a block diagram of a server according to one or more embodiments.

[0040] Figure 5 is a flow chart illustrating a process for processing a request for one or more personalized HRTFs for a user in accordance with one or more embodiments.

[0041] Figure 6 is a block diagram of an audio system according to one or more embodiments.

[0042] Figure 7 is a flow chart illustrating a process for presenting audio content on a head mounted device using one or more personalized HRTFs according to one or more embodiments.

[0043] Figure 8A system environment of a head mounted device including an audio system according to one or more embodiments.

[0044] Fig. 9 is a perspective view of a head mounted device including an audio system according to one or more embodiments.

[0045] The drawings depict various examples for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative examples of the structures and methods shown herein may be employed without departing from the principles described herein.

[0046] Detailed Description

[0047] Overview

[0048] The system environment is configured to generate personalized HRTFs. HRTFs characterize the sound received at a person's ear for a specific location of a sound source. Multiple HRTFs are used to characterize how a user perceives sound. Based on the person's anatomy (e.g., ear shape, shoulders, etc.), the HRTF relative to a person's specific source direction may be unique to the person because their anatomy affects how sound reaches the person's ear canal.

[0049] A typical HRTF specific to a user includes features (e.g., notches) for customizing the HRTF for the user. A template HRTF is an HRTF determined using data from a certain population, which can then be personalized for a single user. Therefore, a single template HRTF is customizable to provide different personalized HRTFs for different users. The template HRTF can be considered as a smoothly varying continuous energy function without a separate source direction frequency characteristic over one or more frequency ranges (e.g., 5kHz-10kHz). A personalized HRTF is generated using the template HRTF by applying one or more filters to the template HRTF. For example, a filter can be used to introduce one or more notches in the template HRTF. In some embodiments, for a given source direction, the notch is described by the following parameters: frequency position, bandwidth centered at the frequency position, and band attenuation value at the frequency position. Notches can be considered as the result of resonance in the sound energy when the sound energy reaches the listener's head and bounces around the head and auricle, undergoing cancellation before reaching the entrance of the ear canal. As described above, notches can affect how a person perceives sound (e.g., what height the sound sounds from relative to the user).

[0050] The system environment includes a server and an audio system (which may be implemented in whole or in part as part of a head-mounted device, may be independent and outside the head-mounted device, etc.). The server may receive acoustic feature data describing the characteristics of a user's head and / or head-mounted device. For example, the user may provide the server system with images and / or videos of their heads and / or ears, anthropometric features of the heads and / or ears, etc. The server determines parameter values ​​of one or more personalized filters (e.g., adding notches) based at least in part on the acoustic feature data. For example, the server may utilize machine learning to identify parameter values ​​of one or more notch filters based on the received acoustic feature data. The server generates one or more personalized HRTFs for the user based on the template HRTF and the personalized filter (e.g., parameter values ​​determined for one or more personalized notches). In some embodiments, the server provides one or more personalized HRTFs to an audio system (e.g., which may be part of a head-mounted device) associated with the user. The audio system may apply one or more personalized HRTFs to the audio data to render the audio data as audio content. The audio system may then present the audio content (e.g., through a speaker component of the audio system). The rendered audio content is spatialized audio content (ie, sounds as if originating from one or more target sound source directions).

[0051] In some embodiments, some or all of the functions of the server are performed by the audio system. For example, the server can provide a personalized filter (e.g., parameter values ​​of one or more personalized notches) to the audio system on the head mounted device, and the audio system can generate one or more personalized HRTFs using the personalized filter and the template HRTF.

[0052] Figure 1 1 is a perspective view of the auditory perception of user 110 when perceiving audio content according to one or more embodiments. An audio system (not shown) presents the audio content to user 110 of the audio system. In this illustrative example, user 110 is placed at the origin of a spherical coordinate system, more specifically, the midpoint between the ears of user 110. When the audio system in a head mounted device provides audio content to user 110, to facilitate an immersive experience for the user, the audio system may spatially position the audio content so that the user perceives the audio content as originating from a source direction 120 relative to the head mounted device. Source direction 120 may be represented by an elevation angle. and azimuth θ140. The elevation angle is the angle measured from the horizontal plane 150 toward the pole of the spherical coordinate system. The azimuth angle is measured from the reference axis on the horizontal plane 150. In other embodiments, the perceived sound origin direction may include one or more vectors, such as a vector angle describing the width of the perceived sound origin direction or a vector solid angle describing the area of ​​the perceived sound origin direction. Using the physical principle that sound pressure decreases with distance r at a ratio of 1 / r, audio content can be further spatially localized as originating from a specific distance in the direction of the target sound source.

[0053] Two parameters that affect sound localization are the user's interaural time differences (ITD) and interaural level differences (ILD). ITD describes the difference in the arrival time of sound between the two ears, which provides a hint of the angle or direction of the sound source from the head. For example, the sound from a sound source located on the right side of a person will reach the right ear before reaching the left ear of the person. ILD describes the difference in sound level or intensity between the two ears. For example, the sound from a sound source located on the right side of a person will be louder when heard in the right ear of the person than in the left ear, because the head blocks part of the sound wave when it propagates to the left ear. ITD and ILD may affect the lateralization of the sound.

[0054] In some embodiments, the user's personalized HRTF is parameterized based on the sound source elevation and azimuth. and a target user audio perception of a particular source direction 120 at a defined value of azimuth angle θ 140, the audio content provided to the user may be modified by a set of HRTFs personalized for the user and for the target source direction 120. Some embodiments may also spatially position the presented audio content at a target distance in the target sound source direction based on the distance between the user 110 and a target location from which the sound is intended to be perceived to originate.

[0055] Template HRTF

[0056] Template HRTF is a customizable HRTF so that it can be personalized for different users. Template HRTF can be considered as a smoothly varying continuous energy function that does not have a separate sound source direction frequency characteristic, but describes the average sound source direction frequency characteristic of a group of listeners (e.g., all listeners in some cases).

[0057] In some embodiments, the template HRTF is generated from a common HRTF for a user group. In some embodiments, the common HRTF corresponds to an average HRTF obtained in the user group. In some embodiments, the common HRTF corresponds to an HRTF in a database of HRTFs obtained from the user group. In some embodiments, the criterion for selecting this one HRTF from the database of HRTFs corresponds to a predefined machine learning or statistical model or statistical measure. The common HRTF exhibits average frequency characteristics of different sound source directions in the user group.

[0058] In some embodiments, the template HRTF can be considered to retain the average angle-dependent ITD and ILD of a general user population. However, the template HRTF does not exhibit any personalized frequency characteristics (e.g., notches in specific locations). Notches can be considered to be the result of resonances in the sound energy when it reaches the listener's head and bounces around the head and pinna, undergoing cancellation before reaching the entrance to the ear canal. The notches in the HRTF (e.g., the number of notches, the location of the notches, the width of the notches, etc.) are customized / personalized HRTFs for specific users. Therefore, the template HRTF is a generic, non-personalized parameterized frequency transfer function that has been modified to remove personalized notches in the spectrum, particularly those between 5kHz and 10kHz. And in some embodiments, these notches may be located below 5kHz and above 10kHz.

[0059] A fully personalized "real" HRTF for a user is a high-dimensional dataset that relies on tens of thousands of parameters to provide a realistic perception of the elevation of sound sources to the listener. Features such as the geometry of the user's head, the shape of the ear pinna, the geometry of the ear canal, the density of the head, environmental characteristics, etc. all transform the audio content as it propagates from the source location and affect how the individual user perceives the audio (e.g., attenuating or amplifying the frequencies of the generated audio content). In short, a personalized "real" HRTF for a user includes personalized notches in the frequency spectrum.

[0060] Figure 2An example depiction of three HRTFs parameterized by a user's sound source elevation angle according to one or more embodiments is shown. The three HRTFs include a real HRTF 210 for the user, a template HRTF 220, and a personalized HRTF 230. The three HRTFs depict color-scale coded energy values ​​in decibels (energy (dB) in the range of -20 dB to 20 dB) for elevation angles in degrees (elevation angles (degrees) in the range of -90 degrees to 90 degrees) parameterized over a set of frequency values ​​in kilohertz (frequency (kHz) in the range of 0.0 kHz to 16.0 kHz), as will be discussed further below. Note that, although not shown, each of these HRTFs has a graph as a function of azimuth angle.

[0061] The real HRTF 210 describes a real frequency attenuation characteristic that affects how the ear receives sound from a point in space within the shown elevation range. Note that within the frequency range of approximately 5.0 kHz-16.0 kHz, the real HRTF 210 exhibits a frequency attenuation characteristic across the entire elevation range. This is visually depicted as a notch 240. This means that for audio content within the frequency band range of 5.0 kHz-16 kHz, in order for the audio content to provide the user with a real immersive experience with respect to the elevation of the sound source, the generated audio content can ideally be convolved with an HRTF that is as close as possible to the real HRTF 210 for the shown elevation range.

[0062] The template HRTF 220 represents an example of frequency attenuation characteristics displayed by a generic centroid HRTF that preserves the average angle-dependent ITD and ILD for a general user population. Note that the template HRTF 220 exhibits similar characteristics to the true HRTF 210 in the frequency range of approximately 0.0 kHz-5.0 kHz. However, in the frequency range of approximately 5.0 kHz-16.0 kHz, unlike the true HRTF 210, the template HRTF 220 exhibits reduced frequency attenuation characteristics in the elevation angle range shown.

[0063] The personalized HRTF 230 is a version of the template HRTF 220 that has been personalized for the user. Figure 3-Figure 7As discussed, personalization applies one or more filters to the template HRTF. One or more filters may be used to introduce one or more notches into the template HRTF. In the illustrated example, two notches 250 are added to the template HRTF 220 to form the personalized HRTF 230. Note that, in part because the notches 250 approximate the notches 240 in the real HRTF 210, the personalized HRTF 230 exhibits similar characteristics to the real HRTF 210 in the frequency range of 0.0 kHz-16.0 kHz.

[0064] System Overview

[0065] Figure 3 3 is a schematic diagram of a high-level system environment 300 for determining a personalized HRTF for a user 310 according to one or more embodiments. A head mounted device 320 communicates with a server 330 via a network 340. The user 310 may wear the head mounted device 320.

[0066] The server 330 receives the acoustic feature data. For example, the user 310 may provide the acoustic feature data to the server 330 via the network 340. The acoustic feature data describes the features of the head and / or the head mounted device 320 of the user 310. The acoustic feature data may include, for example, one or more images of the head and / or ears of the user 310, one or more videos of the head and / or ears of the user 310, anthropometric features of the head and / or ears of the user 310, one or more images of the head wearing the head mounted device 320, one or more images of the head wearing the head mounted device 320 alone, one or more videos of the head wearing the head mounted device 320, one or more videos of the head wearing the head mounted device 320 alone, or some combination thereof. The anthropometric features of the user 310 are the measurements of the head and / or ears of the user 310. In some embodiments, the anthropometric features may be measured using a measuring instrument such as a measuring tape and / or a ruler. In some embodiments, an image and / or video of the head and / or ears of the user 310 is captured using an imaging device (not shown). The imaging device may be a camera on the head mounted device 320, a depth camera assembly (DCA) that is part of the head mounted device 320, an external camera (e.g., part of a mobile device), an external DCA, some other device configured to capture images and / or depth information, or some combination thereof. In some embodiments, the imaging device is also used to capture images of the head mounted device 320. The data may be provided to the server 330 via the network 340.

[0067] In order to more accurately capture the user's head, the user 310 (or some other party) positions the imaging device at different positions relative to their head so that the captured image covers different parts of the user 310 head. The user 310 can hold the imaging device at different angles and / or distances relative to the user 310. For example, the user 310 can hold the imaging device at an arm's length directly in front of the user's 310 face and use the imaging device to capture an image of the user's 310 face. The user 310 can also hold the imaging device at a distance shorter than the arm's length, with the imaging device pointing to the side of the user's 310 head to capture images of the user's 310 ears and / or shoulders. In some embodiments, the imaging device can run feature recognition software and automatically capture images when features of interest (e.g., ears, shoulders) are recognized or input is received from the user to capture an image. In some embodiments, the imaging device can have an application with a graphical user interface (GUI) that guides the user 310 to capture multiple images of the user's 310 head from a specific angle and / or distance relative to the user 310. For example, the GUI may request a frontal image of the face of user 310, an image of the right ear of user 310, and an image of the left ear of user 310. In some embodiments, anthropometric features are determined by an imaging device using images and / or videos captured by the imaging device.

[0068] In the example shown, data is provided from head mounted device 320 to server 330 via network 340. However, in alternative embodiments, some other device (e.g., a mobile device (e.g., a smartphone, tablet, etc.), a desktop computer, an external camera, etc.) may be used to upload data to server 330. In some embodiments, data may be provided directly to server 330.

[0069] Network 340 may be any suitable communication network for data transmission. Network 340 is typically the Internet, but may be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network. In some example embodiments, network 340 is the Internet, and standard communication technologies and / or protocols are used. Therefore, network 340 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, PCI Express (PCI Express) Advanced Switching, etc. In some example embodiments, entities use customized and / or dedicated data communication technologies to replace or supplement the above technologies.

[0070] The server 330 uses the acoustic feature data of the user and the template HRTF to generate a personalized HRTF for the user 310. In some embodiments, there is a single template HRTF for all users. However, in an alternative embodiment, there are multiple different template HRTFs, and each template HRTF is for a different group with one or more common features (e.g., head size, ear shape, male, female, etc.). In some embodiments, each template HRTF is associated with a specific feature. These features can be, for example, head size, head shape, ear size, gender, age, some other feature that affects how a person perceives sound, or some combination thereof. For example, based on changes in head size and / or age, there may be different HRTFs (e.g., there may be template HRTFs for children and different HRTFs for adults) because the ITD may scale with the head diameter. In some embodiments, the server 330 uses the acoustic feature data to determine one or more features (e.g., ear size, shape, head size, etc.) that describe the head of the user 310. The server 330 can then select a template HRTF based on one or more features.

[0071] The server 330 uses a trained machine learning system on the acoustic feature data to obtain a filter customized for the user. The filter can be applied to the template HRTF to create a personalized HRTF. The filter can be, for example, a bandpass (e.g., describing a peak), a bandstop (e.g., describing a notch), a high pass (e.g., describing a high frequency shelf), a low pass (e.g., describing a low frequency shelf), or some combination thereof. The filter can be described by one or more parameter values. The parameter value can include, for example, a frequency position, a bandwidth centered on the frequency position (e.g., determined by a quality factor and / or a filter order), and a depth (e.g., gain) at the frequency position. The depth at the frequency position refers to the attenuation value in the frequency band at the frequency position. A single filter or a combination of filters can be used to describe one or more notches. In some embodiments, the server 330 uses a trained machine learning (ML) model to determine the filter parameter values ​​of one or more personalized filters using the acoustic feature data of the user 310. The ML model can determine the filter based in part on the ITD and / or ILD estimated from the acoustic feature data. As described above, ITD may affect, for example, elevation, and ILD may have some effect on lateralization. Based on the corresponding filter parameter values, one or more personalized filters are each applied to the template HRTF to modify the template HRTF (e.g., add one or more notches) to generate a personalized HRTF for the user 310 (e.g., at least one for each ear). The personalized HRTF can be parameterized by elevation and azimuth. In some embodiments, when multiple users can operate the head-mounted device 320, the ML model can determine the parameter values ​​of the personalized notches to be applied to the template HRTF for each specific individual user so as to generate a personalized HRTF for each of the multiple users.

[0072] In some embodiments, the server 330 provides the personalized HRTF to the head mounted device 320 via the network 340. The audio system (not shown) in the head mounted device 320 stores the personalized HRTF. The head mounted device 320 can then use the personalized HRTF to present audio content to the user 310 so that it sounds originating from a specific location toward the user (e.g., in front of, behind, etc., a virtual object in a room). For example, the head mounted device 320 can convolve the audio data with one or more personalized HRTFs to generate spatialized audio content that, when presented, sounds originating from a specific location (i.e., spatialized audio content).

[0073] In some embodiments, the server 330 provides the generated personalized set of filter parameter values ​​to the head mounted device 320. In this embodiment, an audio system (not shown) in the head mounted device 320 applies the personalized set of filter parameter values ​​to the template HRTF to generate one or more personalized HRTFs. The template HRTFs may be stored locally on the head mounted device 320 and / or retrieved from some other location (e.g., the server 330).

[0074] Figure 4 4 is a block diagram of a server 400 according to one or more embodiments. Server 330 is an embodiment of server 400. Server 400 includes various components, including, for example, a data storage 410, a communication module 420, a template HRTF generation module 430, and an HRTF personalization module 440. Some embodiments of server 400 have different components than those described herein. Similarly, functions may be allocated between components in a manner different from that described herein. And in some embodiments, one or more functions of server 400 may be performed by other components (e.g., an audio system of a head mounted device).

[0075] The data storage 410 stores data for use by the server 400. The data in the data storage 410 may include, for example, one or more template HRTFs, one or more personalized HRTFs, personalized filters (e.g., personalized sets of filter parameter values), user profiles, acoustic feature data, other data related to the use of the server system, audio data, or some combination thereof. In some embodiments, the data storage 410 stores one or more template HRTFs from the template HRTF generation module 430, stores personalized HRTFs from the HRTF personalization module 440, stores personalized sets of filter parameter values ​​from the HRTF personalization module 440, or some combination thereof. In some embodiments, the data storage 410 may periodically receive and store updated template HRTFs with timestamps from the template HRTF generation module 440. In some embodiments, the user's periodically updated personalized HRTF may be received from the HRTF personalization module 440, time-stamped, and stored in the data storage 410. In some embodiments, the data storage 410 may receive and store personalized sets of filter parameter values ​​with timestamps from the HRTF personalization module 440.

[0076] The communication module 420 communicates with one or more head-mounted devices (e.g., the head-mounted device 320). In some embodiments, the communication module 420 can also communicate with one or more other devices (e.g., imaging devices, smart phones, etc.). The communication module 420 can communicate via, for example, the network 340 and / or some direct coupling (e.g., Universal Serial Bus (USB), WIFI, etc.). The communication module 420 can receive a request for a personalized HRTF for a specific user from the head-mounted device, acoustic feature data (from the head-mounted device and / or some other device), or some combination thereof. The communication module 420 can also provide one or more personalized HRTFs, one or more personalized sets of filter parameter values, one or more template HRTFs, or some combination thereof to the head-mounted device.

[0077] The template HRTF generation module 430 generates a template HRTF. The generated template HRTF can be stored in the data storage 410 and can also be sent to the head-mounted device to be stored in the head-mounted device. In some embodiments, the HRTF generation module 430 generates a template HRTF from a universal HRTF. Universal HRTFs are associated with some user groups and may include one or more notches. Notches in universal HRTFs correspond to amplitude changes in frequency windows or frequency bands. Notches are described by the following parameters: frequency position, frequency band width centered at the frequency position, and frequency band attenuation value at the frequency position. In some embodiments, notches in HRTFs are identified as frequency positions where amplitude changes exceed a predetermined threshold. Therefore, notches in universal HRTFs can be considered to represent average attenuation characteristics as a function of frequency and direction for user groups.

[0078] The template HRTF generation module 430 removes the notch in the universal HRTF over some or all of the entire audible frequency band (the range of sounds that humans can perceive) to form a template HRTF. The template HRTF generation module 430 can also smooth the template HRTF so that some or all of it is a smooth and continuous function. In some embodiments, the template HRTF is generated as a smooth and continuous function that lacks notches in some frequency ranges, but does not necessarily lack notches outside those frequency ranges. In some embodiments, the template HRTF makes it so that there are no notches in the frequency range of 5kHz-10kHz. This may be important because the notches in this frequency range vary between different users. This means that in the frequency range of about 5kHz-10kHz, the number of notches, the size of the notches, and the position of the notches may have a great impact on how the sound energy is received at the entrance of the ear canal (thus affecting user perception). Therefore, making the template HRTF as a smooth and continuous function without notches in the frequency range of about 5kHz-10kHz makes it a suitable template that can be personalized for different users. In some embodiments, the template HRTF generation module 430 generates the HRTF template as a smooth and continuous function that lacks notches in all frequency ranges. In some embodiments, the template HRTF generation module 430 generates an HRTF that is a smooth and continuous function over one or more frequency bands, but may include notches outside of the one or more frequency bands. For example, the template HRTF generation module 430 may generate a template HRTF that lacks notches over a frequency range (e.g., approximately 5kHz-10kHz) but may include one or more notches outside of the range.

[0079] Please note that the generic HRTF used to generate the template HRTF is based on a user group. In some embodiments, a group can be selected so that it represents most users, and a single template HRTF is generated from the group and used to generate some or all personalized HRTFs.

[0080] In other embodiments, multiple groups are used to generate different universal HRTFs, and these groups are such that each is associated with one or more common features. These features can be, for example, head size, head shape, ear size, ear shape, age, gender, some other feature that affects how a person perceives sound, or some combination thereof. For example, one group can be used for adults, one group for children, one group for men, one group for women, and so on. The template HRTF generation module 430 can generate a template HRTF for one or more of the multiple universal HRTFs. Therefore, there may be multiple different template HRTFs, and each template HRTF is for a different group that shares some common feature set.

[0081] In some embodiments, the template HRTF generation module 430 may periodically generate new template HRTFs and / or modify previously generated template HRTFs as more population HRTF data is obtained. The template HRTF generation module 430 may store each newly generated template HRTF and / or each update of the template HRTF in the data storage 410. In some embodiments, the server 400 may send the newly generated template HRTF and / or the update of the template HRTF to the head mounted device.

[0082] The HRTF personalization module 440 determines a filter personalized for the user based at least in part on acoustic feature data associated with the user. The filter may include, for example, one or more filter parameter values ​​personalized for the user. The HRTF personalization module 440 uses a trained machine learning (ML) model for the acoustic feature data of the user to determine personalized filter parameter values ​​(e.g., filter parameter values) of one or more personalized filters (e.g., notches) customized for the user. In some embodiments, the personalized filter parameter values ​​are parameterized by the elevation and azimuth angles of the sound source. The ML model is first trained using data collected from a user population. The collected data may include, for example, image data, anthropometric features, and acoustic data. Training may include supervised or unsupervised learning algorithms, including but not limited to linear and / or logistic regression models, neural networks, classification and regression trees, k-means clustering, vector quantization, or any other machine learning algorithm. Acoustic data may include HRTFs measured using an audio measurement device and / or simulated by numerical analysis from a three-dimensional scan of the head.

[0083] In some embodiments, filters and / or filter parameter values ​​are derived directly from the user's image data via machine learning, and the image data corresponds to a single or multiple snapshots of the left and right ears taken by a camera (in a phone or otherwise). In some embodiments, filters and / or filter parameter values ​​are derived from a single or multiple videos of the left and right ears captured by a camera (in a phone or otherwise) by machine learning. In some embodiments, filters and / or filter parameter values ​​are derived from the user's anthropometric features and correspond to the physical features of the left and right ears. These anthropometric features include the height of the left and right ears, the width of the left and right ears, the height of the concha of the left and right ears, the width of the concha of the left and right ears, the height of the cymba of the left and right ears, the height of the ear fossa of the left and right ears, the height and width of the auricle of the left and right ears, the width of the ear intertragal incisure of the left and right ears, and other related physical measurements. In some embodiments, filters and / or filter parameter values ​​are derived from a weighted combination of photographs, videos, and anthropometric measurements.

[0084] In some embodiments, the ML model uses a convolutional neural network model with node layers, where the value at the node of the current layer is a transformation of the value at the node of the previous layer. The transformation in the model is determined by a set of weights and parameters connecting the current layer and the previous layer. In some examples, the transformation can also be determined by a set of weights and parameters used to transform between previous layers in the model.

[0085] The input to the neural network model can be some or all of the user's acoustic feature data and the template HRTF encoded onto the first convolutional layer, and the output of the neural network model is the filter parameter values ​​of one or more personalized notches to be applied to the template HRTF, which are parameterized by the user's elevation and azimuth angles; this is decoded from the output layer of the neural network. The weights and parameters of the transformations across multiple layers of the neural network model can indicate the relationship between the information contained in the starting layer and the information obtained from the final output layer. For example, the weights and parameters can be quantifications of user features, etc., included in the information of the user's image data. The weights and parameters can also be based on historical user data.

[0086] The ML model may include any number of machine learning algorithms. Some other ML models that may be used are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In some embodiments, the ML model includes a deterministic method that has been trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve the quality of a personalized set of filter parameter values ​​generated using measurements from a monitoring system within an audio system at the head mounted device.

[0087] The HRTF personalization module 440 selects an HRTF template for generating one or more personalized HRTFs for the user. In some embodiments, the HRTF personalization module 440 simply retrieves a single HRTF template (e.g., from the data storage 410). In other embodiments, the HRTF personalization module 440 determines one or more features associated with the user from the acoustic feature data and selects a template HRTF from a plurality of template HRTFs using the determined one or more features.

[0088] The HRTF personalization module 440 generates one or more personalized HRTFs for the user using the selected template HRTF and one or more personalized filters (e.g., a set of filter parameter values). The HRTF personalization module 440 applies the personalized filters (e.g., one or more sets of personalized filter parameter values) to the selected template HRTF to form a personalized HRTF. In some embodiments, the HRTF personalization module 440 adds at least one notch to the selected template HRTF using at least one of the one or more personalized filters to generate a personalized HRTF. In this way, the HRTF personalization module 440 is able to approximate a real HRTF (e.g., as described above with respect to the user) by adding one or more notches (personalized to the user) to the template HRTF. Figure 2 In some embodiments, the HRTF personalization module 440 may then provide one or more personalized HRTFs to the head mounted device (via the communication module 420). In an alternative embodiment, the HRTF personalization module 440 provides the personalized set of filter parameter values ​​to the head mounted device, and the head mounted device generates one or more personalized HRTFs using the template HRTF.

[0089] Figure 5 is a flow chart illustrating a process 500 for processing a request for one or more personalized HRTFs for a user according to one or more embodiments. Figure 5 The process is performed by a server (e.g., server 400). In other embodiments, other entities (e.g., a console) may perform some or all steps of the process. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0090] The server 400 receives 510 acoustic feature data associated with the user. For example, the server 400 may receive one or more images of the user's head and / or ears. The acoustic feature data may be provided to the server over a network from, for example, an imaging device, a mobile device, a head-mounted device, etc.

[0091] The server 400 selects 520 a template HRTF. The server 400 selects a template HRTF from one or more templates (e.g., stored in a data store). In some embodiments, the server 400 selects a template HRTF based in part on acoustic feature data associated with the user. For example, the server 400 may determine that the user is a child using the acoustic feature data and select a template HRTF associated with a child (vs. an adult).

[0092] The server 400 determines 530 one or more personalized filters based in part on the acoustic feature data. The determination is performed using a trained machine learning model. In some embodiments, at least one personalized filter describes one or more sets of filter parameter values. Each set of filter parameter values ​​describes a single notch. The personalized filter parameter values ​​describe the frequency position, the bandwidth centered on the frequency position (e.g., determined by the quality factor and / or the filter order), and the depth (e.g., gain) at the frequency position. In some embodiments, the personalized filter parameter values ​​are parameterized for each elevation and azimuth pair in a spherical coordinate system centered on the user. In some embodiments, the personalized filter parameter values ​​are described as being within one or more specific frequency ranges (e.g., 5kHz-10kHz).

[0093] The server 400 generates 540 one or more personalized HRTFs for the user based on the template HRTF and one or more personalized filters (e.g., one or more sets of filter parameter values). The server 400 adds at least one notch to the template HRTF using one or more personalized filters (e.g., through one or more sets of filter parameter values) to generate a personalized HRTF.

[0094] The server 400 provides 550 one or more personalized HRTFs to an audio system associated with the user. In some embodiments, some or all of the audio system may be part of a head mounted device. In other embodiments, some or all of the audio system may be separate from and external to the head mounted device. The audio system may use the one or more personalized HRTFs to render audio content to the user.

[0095] Note that in an alternative embodiment, the server 400 provides one or more personalized filters (and possibly the template HRTF) to the head mounted device, and step 540 is performed by the head mounted device.

[0096] Figure 6 is a block diagram of an audio system 600 according to one or more embodiments. In some embodiments, Figure 6 The audio system 600 is a component of a head mounted device that provides audio content to a user. In other embodiments, some or all of the audio system 600 is separate from the head mounted device and is external to the head mounted device. For example, the audio system 600 can be part of a console. The audio system 600 includes a speaker assembly 610 and an audio controller 620. Some embodiments of the audio system 600 have different components than those described here. Similarly, functionality can be allocated between components in a manner different than that described here.

[0097] The speaker assembly 610 provides audio content to the user of the audio system 600. The speaker assembly 610 includes a speaker that provides audio content according to instructions from the audio controller 620. In some embodiments, one or more speakers of the speaker assembly 610 can be located away from the head-mounted device (for example, in a local area of ​​the head-mounted device). The speaker assembly 610 is configured to provide audio content to one or both ears of the user of the audio system 600 using a speaker. The speaker can be, for example, a moving coil transducer, a piezoelectric transducer, some other device that uses an electrical signal to generate sound pressure waves, or some combination thereof. A typical moving coil transducer includes a coil and a permanent magnet that generates a permanent magnetic field. When the wire is placed in a permanent magnetic field, applying a current to the wire generates a force on the coil according to the amplitude and polarity of the current, which can move the coil toward or away from the permanent magnet. The piezoelectric transducer includes a piezoelectric material that can be strained by applying an electric field or voltage to the piezoelectric material. Some examples of piezoelectric materials include polymers (e.g., polyvinyl chloride (PVC), polyvinylidene fluoride (PVDF)), polymer-based composites, ceramics, or crystals (e.g., quartz (silicon dioxide or SiO2), lead zirconate titanate (PZT)). One or more speakers placed near the user's ears can be coupled to a soft material (e.g., silicone) that adheres well to the user's ears and may be comfortable to the user.

[0098] The audio controller 620 controls the operation of the audio system 600. In some embodiments, the audio controller 620 obtains acoustic feature data associated with the user of the head mounted device. The acoustic feature data can be obtained from an imaging device (e.g., a depth camera assembly) on the head mounted device or from some other device (e.g., a smart phone). In some embodiments, the audio controller 620 can be configured to determine anthropometric features based on data from an imaging device and / or other devices. For example, the audio controller 620 can derive anthropometric features using a weighted combination of photos, videos, and anthropometric results. In some embodiments, the audio controller 620 provides the acoustic feature data to a server (e.g., server 400) via a network (e.g., network 340).

[0099] The audio system 600 generates audio content using one or more personalized HRTFs. One or more personalized HRTFs are customized for the user. In some embodiments, some or all of the one or more personalized HRTFs are received from a server. In some embodiments, the audio controller 620 generates one or more personalized HRTFs using data received from the server (e.g., a personalized set of notch parameters and a template HRTF).

[0100] In some embodiments, the audio controller 620 can identify an opportunity to present audio content with a target sound source direction to a user of the audio system 600, for example, when a flag for presenting audio content with a target sound source direction appears in the virtual experience. The audio controller 620 can first retrieve audio data, which will then be rendered to generate audio content presented to the user. The audio data can additionally specify a target sound source direction and / or target position of a virtual source of audio content within a local area of ​​the audio system 600. Each target sound source direction describes the spatial direction of a virtual source of the sound. In addition, the target sound source position is the spatial position of the virtual source. For example, the audio data can include an explosion from a first target sound source direction and / or target position behind the user, and a bird call from a second target sound source direction and / or target position in front of the user. In some embodiments, the target sound source directions and / or target positions can be organized in a spherical coordinate system, with the user located at the origin of the spherical coordinate system. Each target sound source direction is then represented as an elevation angle relative to the horizontal plane and an azimuth angle in the spherical coordinate system, such as Figure 1 The target sound source position includes the elevation angle from the horizontal plane, the azimuth angle and the distance from the origin of the spherical coordinate system.

[0101] The audio controller 620 uses one or more personalized HRTFs for the user based on the target audio source direction and / or position awareness associated with the audio data to be presented to the user. The audio controller 620 convolves the audio data with the one or more personalized HRTFs to render audio content to the user that is spatialized to sound as originating from the target source direction and / or position. The audio controller 620 provides the rendered audio content to the speaker assembly 610 for presentation to the user of the audio system.

[0102] Figure 7 7 is a flow chart illustrating a process 700 for presenting audio content on a head mounted device using one or more personalized HRTFs according to one or more embodiments. Figure 7 The process of is performed by the head mounted device. In other embodiments, other entities may perform some or all steps of the process. For example, steps 710 and 720 may be performed by some other device. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0103] The head mounted device captures 710 acoustic feature data of the user. The head mounted device may capture images and / or videos of the user's head and ears, for example, using an imaging device in the head mounted device. In some embodiments, the head mounted device may communicate with an external device (e.g., a camera, a mobile device / phone, etc.) to receive the acoustic feature data.

[0104] The head mounted device provides 720 the acoustic feature data to a server (e.g., a server system). In some embodiments, the acoustic feature data may be pre-processed at the head mounted device before being provided to the server. For example, in some embodiments, the head mounted device may use the captured images and / or videos to determine anthropometric features of the user.

[0105] The head mounted device receives 730 one or more personalized HRTFs from the server. The one or more personalized HRTFs are customized for the user.

[0106] The head mounted device renders 740 audio content using one or more personalized HRTFs. The head mounted device may convolve the audio data with the one or more personalized HRTFs to generate the audio content. The audio content is rendered by a speaker assembly and is perceived as originating from a target source direction and / or target location.

[0107] In the above embodiment, the server provides the personalized HRTF to the head mounted device. However, in an alternative embodiment, the server may provide the template HRTF, one or more personalized filters (e.g., one or more sets of personalized filter parameter values), or some combination thereof to the head mounted device. The head mounted device will then generate the personalized HRTF using the one or more personalized filters.

[0108] Artificial reality system environment

[0109] Figure 8 8 is a system environment 800 including a head mounted device 805 of an audio system 600 according to one or more embodiments. The system 800 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 8 The illustrated system 800 includes a head mounted device 805 and an input / output (I / O) interface 815 coupled to a console 810, and the console 810 and / or the head mounted device 805 communicate with the server 400 via a network 340. The head mounted device 805 may be an embodiment of the head mounted device 320. Figure 8 An example system 800 is shown that includes one head mounted device 805 and one I / O interface 815, but in other embodiments, any number of these components may be included in the system 800. For example, there may be multiple head mounted devices 805, each with an associated I / O interface 815, each head mounted device 805 and I / O interface 815 communicating with the console 810. In alternative configurations, different and / or additional components may be included in the system 800. Additionally, in some embodiments, the system 800 may include a plurality of head mounted devices 805, each with an associated I / O interface 815, each head mounted device 805 and I / O interface 815 communicating with the console 810. Figure 8 The functionality described by one or more components shown may be different in combination. Figure 8 The described approach is distributed among the components. For example, some or all of the functions of the console 810 are provided by the head mounted device 805.

[0110] The head-mounted device 805 may be a near-eye display (NED) or head-mounted display (HMD) that presents content to the wearer, the content including an augmented view of a physical reality environment with computer-generated elements (e.g., two-dimensional (2D) or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In some embodiments, the presented content includes audio presented via an audio system 600, which receives audio information from the head-mounted device 805, the console 810, or both, and presents audio data based on the audio information. In some embodiments, the head-mounted device 805 presents virtual content to the wearer that is based in part on the real environment surrounding the wearer. For example, the virtual content may be presented to the wearer of the head-mounted device. The head-mounted device includes the audio system 600. The head-mounted device 805 may also include a depth camera assembly (DCA) 825, an electronic display 830, an optical block 835, one or more position sensors 840, and an inertial measurement unit (IMU) 845. Some embodiments of the head-mounted device 805 have a configuration associated with the head-mounted device 805. Figure 8 In addition, in other embodiments, by combining Figure 8 The functionality provided by the various components described may be distributed differently among the components of the head mounted device 805, or captured in a separate component remote from the head mounted device 805. Fig. 9 An example of a head mounted device is described.

[0111] The audio system 600 presents audio content to a user of the head mounted device 805 using one or more personalized HRTFs. In some embodiments, the audio system 600 may receive and store the user's personalized HRTF (e.g., from the server 400 and / or the console 810). In some embodiments, the audio system 600 may receive and store a template HRTF and / or one or more personalized filters (e.g., described by parameter values) to be applied to the template HRTF (e.g., from the server 400 and / or the console 810). The audio system 600 receives audio data associated with a target sound source direction relative to the head mounted device 805. The audio system 600 applies one or more personalized HRTFs to the audio data to generate audio content. The audio system 600 presents the audio content to the user through a speaker assembly. The presented audio content is spatialized so that when presented with the speaker assembly, it sounds originating from the target sound source direction and / or the target position.

[0112] The DCA 825 captures data describing depth information of a local area around some or all of the head-mounted device 805. The DCA 825 may include a light emitter, an imaging device, and a DCA controller that may be coupled to both the light emitter and the imaging device. The light emitter illuminates the local area with illumination light, for example, according to an emission instruction generated by the DCA controller. The DCA controller is configured to control the operation of specific components of the light emitter based on the emission instruction, for example, to adjust the intensity and pattern of the illumination light illuminating the local area. In some embodiments, the illumination light may include a structured light pattern, such as a dot pattern, a line pattern, etc. The imaging device captures one or more images of one or more objects in the local area illuminated by the illumination light. The DCA 825 may use the data captured by the imaging device to calculate the depth information, or the DCA 825 may send the information to another device (e.g., the console 810), which may use the data from the DCA 825 to determine the depth information. The DCA 825 may also be used to capture depth information describing the user's head and / or ears by removing the head-mounted device and pointing the DCA at the user's head and / or ears.

[0113] The electronic display 830 displays 2D or 3D images to the wearer based on data received from the console 810. In various embodiments, the electronic display 830 includes a single electronic display or multiple electronic displays (e.g., one display for each eye of the wearer). Examples of electronic displays 830 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof.

[0114] The optical block 835 amplifies the image light received from the electronic display 830, corrects the optical errors associated with the image light, and presents the corrected image light to the wearer of the head mounted device 805. In various embodiments, the optical block 835 includes one or more optical elements. Example optical elements included in the optical block 835 include: a waveguide, an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 835 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 835 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0115] The magnification and focusing of the image light by the optical block 835 allows the electronic display 830 to be physically smaller, lighter, and consume less power than a larger display. In addition, the magnification can increase the field of view of the content presented by the electronic display 830. For example, the field of view of the displayed content is such that the displayed content is presented using nearly all of the wearer's field of view (e.g., approximately 110 degrees diagonal), and in some cases all of the field of view. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0116] In some embodiments, the optical block 835 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display 830 for display is pre-distorted, and when the optical block 835 receives image light generated based on the content from the electronic display 830, the optical block 835 corrects the distortion.

[0117] The IMU 845 is an electronic device that generates data indicating the position of the head mounted device 805 based on measurement signals received from one or more position sensors 840. The position sensor 840 generates one or more measurement signals in response to the movement of the head mounted device 805. Examples of the position sensor 840 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 845, or some combination thereof. The position sensor 840 can be located external to the IMU 845, internal to the IMU 845, or some combination of both.

[0118] Based on one or more measurement signals from one or more position sensors 840, the IMU 845 generates data indicating an estimated current position of the head mounted device 805 relative to an initial position of the head mounted device 805. For example, the position sensor 840 includes a plurality of accelerometers for measuring translational motion (forward / backward, up / down, left / right) and a plurality of gyroscopes for measuring rotational motion (e.g., pitch, yaw, and roll). In some embodiments, the IMU 845 samples the measurement signals rapidly and calculates the estimated current position of the head mounted device 805 based on the sampled data. For example, the IMU 845 integrates the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated current position of a reference point on the head mounted device 805. Alternatively, the IMU 845 provides the sampled measurement signals to the console 810, which parses the data to reduce errors. The reference point is a point that can be used to describe the position of the head mounted device 805. A reference point can generally be defined as a point or location in space that is related to the orientation and position of the head mounted device 805.

[0119] The I / O interface 815 is a device that allows the wearer to send an action request and receive a response from the console 810. An action request is a request to perform a specific action. For example, an action request can be an instruction to start or end capturing image or video data, or an instruction to perform a specific action within an application. The I / O interface 815 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 810. The action request received by the I / O interface 815 is transmitted to the console 810, and the console 810 performs the action corresponding to the action request. In some embodiments, as further described above, the I / O interface 815 includes an IMU 845, which captures calibration data indicating the estimated position of the I / O interface 815 relative to the initial position of the I / O interface 815. In some embodiments, the I / O interface 815 can provide tactile feedback to the wearer according to the instructions received from the console 810. For example, tactile feedback is provided when an action request is received, or when console 810 transmits instructions to I / O interface 815 that cause I / O interface 815 to generate tactile feedback when console 810 performs an action.

[0120] The console 810 provides content to the head mounted device 805 for processing based on information received from one or more of: the head mounted device 805 and the I / O interface 815. Figure 8 In the example shown, the console 810 includes an application store 850, a tracking module 855, and an engine 860. Some embodiments of the console 810 have a Figure 8 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Figure 8 The described manner is distributed among the components of the console 810 .

[0121] The application storage 850 stores one or more applications for execution by the console 810. An application is a set of instructions that, when executed by a processor, generates content for presentation to the wearer. The content generated by the application may be in response to input received from the wearer via the movement or I / O interface 815 of the head mounted device 805. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0122] The tracking module 855 calibrates the system environment 800 using one or more calibration parameters, and may adjust the one or more calibration parameters to reduce errors in the determination of the position of the head mounted device 805 or the I / O interface 815. The calibration performed by the tracking module 855 may also take into account information received from the IMU 845 in the head mounted device 805 and / or the IMU 845 included in the I / O interface 815. In addition, if tracking of the head mounted device 805 is lost, the tracking module 855 may recalibrate some or all of the system environment 800.

[0123] The tracking module 855 tracks the movement of the head mounted device 805 or the I / O interface 815 using information from one or more position sensors 840, the IMU 845, the DCA 825, or some combination thereof. For example, the tracking module 855 determines the location of a reference point of the head mounted device 805 in a map of the local area based on the information from the head mounted device 805. The tracking module 855 may also determine the location of the reference point of the head mounted device 805 or the reference point of the I / O interface 815 using data indicating the location of the head mounted device 805 from the IMU 845 or using data indicating the location of the I / O interface 815 from an IMU 845 included in the I / O interface 815, respectively. Additionally, in some embodiments, the tracking module 855 may use partial data indicating the location of the head mounted device 805 from the IMU 845 to predict the future location of the head mounted device 805. The tracking module 855 provides the estimated or predicted future position of the head mounted device 805 or the I / O interface 815 to the engine 860.

[0124] The engine 860 also executes applications within the system environment 800 and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the head mounted device 805 from the tracking module 855. Based on the received information, the engine 860 determines the content to be provided to the head mounted device 805 for presentation to the wearer. For example, if the received information indicates that the wearer has looked to the left, the engine 860 generates content for the head mounted device 805 that reflects (mirror) the movement of the wearer in a virtual environment or in an environment that enhances a local area with additional content. In addition, the engine 860 performs an action within an application executed on the console 810 in response to an action request received from the I / O interface 815, and provides feedback to the wearer that the action is performed. The feedback provided may be visual or auditory feedback via the head mounted device 805, or tactile feedback via the I / O interface 815.

[0125] Example Headset

[0126] Fig. 9 900 including an audio system according to one or more embodiments. The head-mounted device 900 presents media to a user. Examples of media presented by the head-mounted device 900 include one or more images, videos, audio, or some combination thereof. The head-mounted device 900 may be a near-eye display, glasses, or a head-mounted display (HMD). The head-mounted device 900 includes components such as a frame 905, a lens 910, a sensor device 915, and an audio system (not shown). In an embodiment as a head-mounted device, the head-mounted device 900 may correct or enhance the user's vision, protect the user's eyes, or provide images to the user. The head-mounted device 900 may be glasses that correct the user's vision defects. The head-mounted device 900 may be sunglasses that protect the user's eyes from sunlight. The head-mounted device 900 may be safety glasses that protect the user's eyes from impact. The head-mounted device 900 may be a night vision device or infrared goggles to enhance the user's vision at night. In an alternative embodiment, the head mounted device 900 may not include the lenses 910 and may be a frame 905 having an audio system that provides audio content (e.g., music, radio, podcasts) to the user.

[0127] The frame 905 includes a front portion that holds the lenses 910 and an end piece that attaches to the user. The front portion of the frame 905 bridges the top of the user's nose. The end piece (e.g., temple) is the portion of the frame 905 to which the user's temples are attached. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that curls behind the user's ear (e.g., temple tip, ear piece).

[0128] The lens 910 provides or transmits light to the user wearing the head mounted device 900. The lens 910 is held by the front of the frame 905 of the head mounted device 900. The lens 910 may be a prescription lens (e.g., a single vision lens, a bifocal lens, and a trifocal lens or a progressive lens) to help correct the user's vision defects. The prescription lens transmits ambient light to the user wearing the head mounted device 900. The transmitted ambient light may be changed by the prescription lens to correct the user's vision defects. The lens 910 may be a polarized lens or a tinted lens to protect the user's eyes from sunlight. The lens 910 may be one or more waveguides as part of a waveguide display, wherein the image light is coupled to the user's eyes through the end or edge of the waveguide. The lens 910 may include an electronic display for providing image light, and may also include an optical block for amplifying the image light from the electronic display. In some embodiments, the lens 910 is an embodiment of the electronic display 830.

[0129] The sensor device 915 estimates the current position of the head mounted device 900 relative to the initial position of the head mounted device 900. The sensor device 915 may be located on a portion of the frame 905 of the head mounted device 900. The sensor device 915 includes a position sensor and an inertial measurement unit. The sensor device 915 may also include one or more cameras placed on the frame 905 to observe or face the eyes of the user. The one or more cameras of the sensor device 915 are configured to capture image data corresponding to the eye position of the user's eyes. The sensor device 915 may be an embodiment of the IMU 845 and / or the position sensor 840.

[0130] An audio system (not shown) provides audio content to a user of the head mounted device 900. The audio system is an embodiment of the audio system 600, and uses speakers 920 to present content.

[0131] Additional configuration information

[0132] Embodiments according to the invention are specifically disclosed in the appended claims, relating to methods, storage media, and audio systems, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., storage medium, audio system, system, and computer program product). Dependencies or back-references in the appended claims are selected for formal reasons only. However, any subject matter resulting from an intentional back-reference to any preceding claim (in particular multiple dependencies) may also be claimed, such that any combination of claims and their features is disclosed and may be asserted, regardless of the dependencies selected in the appended claims.

[0133] The subject matter that may be claimed includes not only the combination of features as set forth in the appended claims, but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature of the appended claims.

[0134] In one embodiment, a method may include: determining one or more personalized filters based at least in part on acoustic feature data of a user; generating one or more personalized head-related transfer functions (HRTFs) for the user based on a template HRTF and the determined one or more personalized filters; and providing the generated one or more personalized HRTFs to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

[0135] Determining one or more personalized filters may include using a trained machine learning model and the user's acoustic feature data to determine parameter values ​​for one or more personalized filters. The parameter values ​​for one or more personalized filters may describe one or more personalized notches in one or more personalized HRTFs. The parameter values ​​may include: a frequency position, a width in a frequency band centered at the frequency position, and an amount of attenuation caused in a frequency band centered at the frequency position.

[0136] The machine learning model may be trained with image data, anthropometric features, and acoustic data, including measurements of HRTFs obtained for a population of users.

[0137] Generating one or more personalized HRTFs for the user based on the template HRTF and the determined one or more personalized filters may include adding at least one notch to the template HRTF using at least one of the one or more personalized filters to generate the personalized HRTF in the one or more personalized HRTFs.

[0138] The template HRTF may be based on a generic HRTF describing a user group, and the generic HRTF may include at least one notch in a frequency range. The template HRTF may be generated from the generic HRTF by removing at least one notch so that the template HRTF is a smooth and continuous function in the frequency range. The frequency range may be 5kHz to 10kHz. At least one notch may exist in the template HRTF outside the frequency range.

[0139] The audio system may be part of the head mounted device. The audio system may be separate from the head mounted device and be external to the head mounted device.

[0140] In one embodiment, a non-transitory computer-readable medium may be configured to store program code instructions which, when executed by a processor, may cause the processor to perform steps including: determining one or more personalized filters based at least in part on acoustic feature data of a user; generating one or more personalized head-related transfer functions (HRTFs) for the user based on a template HRTF and the determined one or more personalized filters; and providing the generated one or more personalized HRTFs to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

[0141] Determining one or more personalized filters may include using a trained machine learning model and acoustic feature data of the user to determine parameter values ​​of the one or more personalized filters.

[0142] The parameter values ​​of one or more personalized filters may describe one or more personalized notches in one or more personalized HRTFs. The parameter values ​​may include: a frequency position, a width in a frequency band centered at the frequency position, and an amount of attenuation caused in the frequency band centered at the frequency position.

[0143] The machine learning model may be trained with image data, anthropometric features, and acoustic data, including measurements of HRTFs obtained for a population of users.

[0144] Generating one or more personalized HRTFs for the user based on the template HRTF and the determined one or more personalized filters may include adding at least one notch to the template HRTF using at least one of the one or more personalized filters to generate the personalized HRTF in the one or more personalized HRTFs.

[0145] In one embodiment, a method may include: receiving, at a head mounted device, one or more personalized HRTFs for a user of the head mounted device; retrieving audio data associated with a target sound source direction relative to the head mounted device; applying the one or more personalized HRTFs to the audio data to render the audio data as audio content; and presenting the audio content by a speaker assembly of the head mounted device, wherein the presented audio content is spatialized so that it sounds as if it originates from the target sound source direction.

[0146] In one embodiment, a method may include: capturing acoustic feature data of a user; and transmitting the captured acoustic feature data to a server, wherein the server uses the captured acoustic feature data to determine one or more personalized HRTFs, and the server provides the one or more personalized HRTFs to a head mounted device.

[0147] In one embodiment, an audio system may include: an audio component and an audio controller, the audio component including one or more speakers configured to present audio content to a user of the audio system; the audio controller is configured to perform a method according to or within any of the above embodiments.

[0148] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to or within any of the embodiments described above.

[0149] In an embodiment, an audio system and / or system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, when executing the instructions, the processor is operable to perform a method according to or within any of the above embodiments.

[0150] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, when executed on a data processing system, may be operable to perform a method according to or within any of the above embodiments.

[0151] The foregoing description of the embodiments of the present disclosure is presented for illustrative purposes; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art can recognize that many modifications and variations are possible in light of the above disclosure.

[0152] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0153] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer readable medium, the computer readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0154] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0155] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing process, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein. Finally, the language used in the specification is selected primarily for readability and guidance purposes, and it may not be selected to depict or limit the creative subject matter. Therefore, it is intended that the scope of the present disclosure is not limited by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, to the scope of the present disclosure, and the scope of the present disclosure is set forth in the appended claims.

Claims

1. A method for generating spatialized audio content, comprising: determining one or more personalized filters based at least in part on acoustic characteristic data of a user, the acoustic characteristic data describing characteristics of a head of the user and a head mounted device worn by the user; applying the one or more personalized filters to a template HRTF to modify the template HRTF to form one or more personalized HRTFs, wherein each of the one or more personalized HRTFs is a version of the template HRTF customized for the user, the template HRTF includes a smoothly varying function that does not exhibit frequency characteristics personalized for the user within a first frequency band, and at least one of the one or more personalized HRTFs includes frequency characteristics personalized for the user within the first frequency band; and The generated one or more personalized HRTFs are provided to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

2. The method according to claim 1, wherein: Determining the one or more personalized filters includes using a trained machine learning model and acoustic feature data of the user to determine parameter values ​​of the one or more personalized filters.

3. The method according to claim 2, wherein: The parameter values ​​of the one or more personalized filters describe one or more personalized notches in the one or more personalized HRTFs, and the parameter values ​​include one or more of: a frequency position, a width in a frequency band centered on the frequency position, and an amount of attenuation caused in the frequency band centered on the frequency position.

4. The method according to claim 2, wherein: The trained machine learning model is trained using image data, anthropometric features and / or acoustic data, the acoustic data comprising measurements of HRTFs obtained for a population of users.

5. The method according to any one of claims 1 to 4, wherein: Applying the one or more personalized filters to the template HRTF to modify the template HRTF and form the one or more personalized HRTFs includes: At least one notch is added to the template HRTF using at least one of the one or more personalized filters to generate a personalized HRTF among the one or more personalized HRTFs.

6. The method according to any one of claims 1 to 4, wherein: The template HRTF is based on a generic HRTF describing a user group, the generic HRTF comprising at least one notch in a frequency range.

7. The method according to claim 5, wherein: The template HRTF is based on a generic HRTF describing a user group, the generic HRTF comprising at least one notch in a frequency range.

8. The method according to claim 6, wherein: The template HRTF is generated from the generic HRTF by removing the at least one notch so that the template HRTF is a smooth and continuous function within the frequency range, the frequency range being 5 kHz to 10 kHz, and at least one notch exists in the template HRTF outside the frequency range.

9. The method according to claim 7, wherein: The template HRTF is generated from the generic HRTF by removing the at least one notch so that the template HRTF is a smooth and continuous function within the frequency range, the frequency range being 5 kHz to 10 kHz, and at least one notch exists in the template HRTF outside the frequency range.

10. The method according to any one of claims 1-4 and 7-9, wherein: The audio system is part of a head mounted device.

11. The method according to claim 5, wherein: The audio system is part of a head mounted device.

12. The method according to claim 6, wherein: The audio system is part of a head mounted device.

13. The method according to any one of claims 1 to 4 and 7 to 9, wherein: The audio system is separate from and external to the head-mounted device.

14. The method according to claim 5, wherein: The audio system is separate from and external to the head-mounted device.

15. The method according to claim 6, wherein: The audio system is separate from and external to the head-mounted device.

16. A non-transitory computer readable medium configured to store program code instructions which, when executed by a processor, cause the processor to perform steps comprising: determining one or more personalized filters based at least in part on acoustic characteristic data of a user, the acoustic characteristic data describing characteristics of a head of the user and a head mounted device worn by the user; applying the one or more personalized filters to a template HRTF to modify the template HRTF to form one or more personalized HRTFs, wherein each of the one or more personalized HRTFs is a version of the template HRTF customized for the user, the template HRTF includes a smoothly varying function that does not exhibit frequency characteristics personalized for the user within a first frequency band, and at least one of the one or more personalized HRTFs includes frequency characteristics personalized for the user within the first frequency band; and The generated one or more personalized HRTFs are provided to an audio system, wherein the personalized HRTFs are used to generate spatialized audio content.

17. The computer-readable medium of claim 16, wherein: Determining the one or more personalized filters includes using a trained machine learning model and acoustic feature data of the user to determine parameter values ​​of the one or more personalized filters, and training the machine learning model with image data, anthropometric features and / or acoustic data, wherein the acoustic data includes measured values ​​of HRTF obtained for a user group.

18. The computer-readable medium of claim 17, wherein: The parameter values ​​of the one or more personalized filters describe one or more personalized notches in the one or more personalized HRTFs, and the parameter values ​​include one or more of: a frequency position, a width in a frequency band centered on the frequency position, and an amount of attenuation caused in the frequency band centered on the frequency position.

19. The computer-readable medium according to any one of claims 16 to 18, wherein: Applying the one or more personalized filters to the template HRTF to modify the template HRTF and form the one or more personalized HRTFs includes: At least one notch is added to the template HRTF using at least one of the one or more personalized filters to generate a personalized HRTF among the one or more personalized HRTFs.

20. A method for presenting spatialized audio content, comprising: Receiving, at a head mounted device, one or more personalized HRTFs of a user of the head mounted device from a server, and generating, at the server, the one or more personalized HRTFs by applying one or more personalized filters to modify a template HRTF used to form the one or more personalized HRTFs, wherein each of the generated one or more personalized HRTFs is a version of the template HRTF customized for the user, the template HRTF includes a smoothly varying function that does not exhibit frequency characteristics personalized for the user within a first frequency band, and at least one of the one or more personalized HRTFs includes frequency characteristics personalized for the user within the first frequency band; retrieving audio data associated with a target sound source direction relative to the head mounted device; applying the one or more personalized HRTFs to the audio data to render the audio data into audio content; and The audio content is rendered by a speaker assembly of the head mounted device, wherein the rendered audio content is spatialized such that the rendered audio content sounds as if it originates from the target sound source direction.

21. The method according to claim 20, further comprising: capturing acoustic characteristic data of the user; and transmitting the captured acoustic feature data to the server, The server uses the captured acoustic feature data to determine the one or more personalized filters.

Images