Inferring pinna information via beamforming to generate personalized spatial audio

Abstract

The audio system presents spatialized audio content to a user that is individually calibrated for the user. The audio system presents sounds to the user that are reflected off the user's ears. An acoustic sensor array of the audio system generates audio data from the presented sounds. The audio system processes the audio data using beamformers, each beamformer directed to a respective portion of the ear to generate a beamformed signal. The audio system uses the beamformed signals to determine a transfer function that defines a transformation of the sound caused by reflections off the user's ears. The audio system generates spatialized audio content for the ears based on the transfer function.

Description

Background Art

[0001] The present disclosure generally relates to generating spatialized audio content for individual users.

[0002] Based on the shape of the user's ears and other acoustic properties, spatialized audio content may sound different for different users. For each ear, the sound from the sound source is transformed via reflection from the auricle before reaching the ear canal. Using microphones placed at the ears to capture the sound reaching the ears, and calculating filters for how each ear transforms the sound, it is possible to customize audio content that takes into account the sound transformations performed by the user's ears. However, binaural microphones may impair the user's normal hearing and limit the user's perception of their surroundings. In addition, binaural microphones may be socially unacceptable and aesthetically unappealing. Summary of the invention

[0003] Therefore, the present invention discloses methods, systems, and computer-readable non-transitory storage media storing instructions for presenting spatialized audio content.

[0004] Embodiments relate to generating spatialized audio content personalized for a user based on audio data captured by a sensor array of acoustic sensors away from the user's ears. Some embodiments include a method for generating audio content for an ear. The method includes: an acoustic sensor of the sensor array generates audio data from one or more sounds received by the acoustic sensor. The audio data is processed using a beamformer, each beamformer pointing to a corresponding portion of the user's ear (e.g., a different position on the auricle of the ear) to generate a beamformed signal. A transfer function is determined using the beamformed signal, which defines the transformation of the sound caused by the reflection from the portion of the ear. The spatialized audio content of the ear is generated using the transfer function. For example, a transfer function can be used to determine an at-ear equalization filter, and the spatialized audio content can be generated by transforming the audio content for the ear using the at-ear equalization filter. A similar process can be performed for the other ear of the user to generate spatialized audio content for the left and right ears personalized for the user.

[0005] Some embodiments include an audio system that includes a sensor array and an audio controller. The sensor array includes an acoustic sensor configured to generate audio data from one or more sounds received by the acoustic sensor. The audio controller generates a beamformed signal by processing the audio data using a beamformer for the acoustic sensor, each beamformer directed to a corresponding portion of a user's ear. The audio controller uses the beamformed signal to determine a transfer function that defines a transformation of the sound caused by reflections from a portion of the ear and generates audio content for the ear using the transfer function.

[0006] The present invention relates to a method comprising: generating audio data from one or more sounds received by the acoustic sensor of a sensor array, generating beamformed signals by processing the audio data using beamformers, each beamformer being directed to a corresponding part of an ear of a user, determining a transfer function using the beamformed signals, and generating spatialized audio content for the ear based on the transfer function, the transfer function defining a transformation of the sound caused by reflections from a part of the ear.

[0007] In one embodiment of the method according to the present invention, generating the spatialized audio content based on the transfer function may include: determining an earside equalization filter based on the transfer function, and using the earside equalization filter to adjust the audio content for the user.

[0008] In another embodiment of the method according to the invention, determining the ear side equalization filter may comprise referring to a database of reference ear side equalization filters.

[0009] In yet another embodiment of the method according to the invention, determining the earside equalization filter may comprise correlating the transfer function with a filter calibrated for the user.

[0010] In another embodiment of the method according to the present invention, using the beamformed signal to determine a transfer function defining a transformation of a sound caused by a reflection from a part of the ear may include: generating other audio data from one or more other sounds received by the acoustic sensor of the sensor array in the absence of reflections from the part of the ear, generating a calibration signal by processing the other audio data using a beamformer, and determining the transfer function using the beamformed signal and the calibration signal.

[0011] In another embodiment of the method according to the present invention, at least one acoustic sensor of the sensor array can be placed at the entrance of the ear canal of the user's ear, and using the beamformed signal to determine a transfer function defining a transformation of sound caused by reflection from a part of the ear may include: generating other audio data by at least one acoustic sensor of the sensor array from one or more other sounds received by the at least one acoustic sensor, and determining the transfer function using the beamformed signal and the other audio data.

[0012] In yet another embodiment of the method according to the invention, the beamformed signals may collectively be indicative of a sound pressure measurement at the center of the user's ear.

[0013] In another embodiment of the method according to the invention, the method may further comprise generating, by at least one transducer, one or more sounds received by the acoustic sensor.

[0014] In another embodiment of the method according to the invention, each beamformer may be directed towards a different part of the pinna of the ear.

[0015] In another embodiment of the method according to the present invention, the method may also include: generating a first beamformer pointing to a first part of the beamformer toward the ear, generating first audio data of the audio content from a first sound among the one or more sounds by an acoustic sensor of the sensor array, and processing the first audio data using the first beamformer to generate a first beamformed signal among the beamformed signals.

[0016] The present invention further discloses an audio system, which includes a sensor array, the sensor array including an acoustic sensor and an audio controller, the acoustic sensor being configured to generate audio data from one or more sounds received by the acoustic sensor; the audio controller being configured to: generate a beamformed signal by processing the audio data using a beamformer for the acoustic sensor of the sensor array, each beamformer being directed to a corresponding part of a user's ear; use the beamformed signal to determine a transfer function defining a transformation of a sound caused by reflections from a part of the ear; and generate spatialized audio content for the ear based on the transfer function.

[0017] In one embodiment of the system according to the present invention, the audio controller may be further configured to determine an ear-side equalization filter based on the transfer function, and adjust the audio content using the ear-side equalization filter.

[0018] In another embodiment of the system according to the present invention, the audio controller may be further configured to refer to a database of reference ear-side equalization filters.

[0019] In yet another embodiment of the system according to the invention, the audio controller may be further configured to relate the transfer function to the filter calibrated for the user.

[0020] In another embodiment of the system according to the present invention, the audio controller can also be configured to: generate other audio data from one or more other sounds received by the acoustic sensor of the sensor array in the absence of reflections from parts of the ear, generate a calibration signal by processing the other audio data using a beamformer, and determine the transfer function using the beamformed signal and the calibration signal.

[0021] In another embodiment of the system according to the present invention, at least one acoustic sensor of the sensor array can be placed at the entrance of the ear canal of the user's ear, and the audio controller can also be configured to: generate other audio data from one or more other sounds received by the at least one acoustic sensor of the sensor array, and determine the transfer function using the beamformed signal and other audio data.

[0022] In yet another embodiment of the system according to the invention, the beamformed signals may collectively be indicative of a sound pressure measurement at the center of the user's ear.

[0023] In another embodiment of the system according to the invention, each beamformer may be directed towards a different part of the pinna of the ear.

[0024] The present invention also discloses a computer-readable non-transitory storage medium storing instructions for presenting spatialized audio content, which instructions, when executed by a processor, cause the processor to perform steps including: generating audio data from one or more sounds received by the acoustic sensor of the sensor array, generating a beamformed signal by processing the audio data using a beamformer, each beamformer being directed to a corresponding part of a user's ear, determining a transfer function defining a transformation of the sound caused by reflections from a part of the ear, and generating spatialized audio content for the ear based on the transfer function.

[0025] In an embodiment of the computer-readable non-transitory storage medium according to the present invention, the instructions further cause the processor to perform steps including: determining an ear-side equalization filter based on the transfer function, and adjusting the audio content for the user using the ear-side equalization filter. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1A is a perspective view of a headset according to one or more embodiments.

[0027] Figure 1Bis a perspective view of a headset implemented as a head mounted display in accordance with one or more embodiments.

[0028] Figure 2 is a cross-sectional diagram of a user's ear illustrating reflection points on portions of the ear according to one or more embodiments.

[0029] Figure 3 is a block diagram of an example audio system in accordance with one or more embodiments.

[0030] Figure 4 is a flowchart of a process for generating spatialized audio content personalized for a user's ears according to one or more embodiments.

[0031] Figure 5 is a block diagram of an example artificial reality system in accordance with one or more embodiments.

[0032] The accompanying drawings depict various embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0033] An audio system presents spatialized audio content personalized for a user. For each ear, the audio system modifies the audio content using an ear-side equalization filter determined by capturing audio data using an acoustic sensor remote from the user's ear and processing the audio data using a beamformer directed at multiple locations of the pinna of the ear. The spatialized audio content includes audio data that provides spatial cues by being different for the left and right ears. The user perceives the spatialized audio content as if they were physically located near the sound source that produced the audio content because the spatialized audio content includes directionality and other spatial cues.

[0034] To capture binaural audio that has been transformed by the user's ears, the audio system can use binaural acoustic sensors placed at each ear of the user. The difference between the sound at each ear of the user and the sound at the sound source can be used to determine a filter that is used to generate audio content that appears to originate from the direction of the sound source after reflecting from the user's specific ears. However, binaural microphones may prevent the user from being fully aware of their surroundings because the microphones partially or completely block the entrance to the user's ear canal.

[0035] Embodiments include an audio system that generates spatialized audio content without using binaural microphones by determining an ear-side equalization filter. The audio system uses a beamformer directed toward a specific portion of the pinna of a user's ear. The audio system monitors how sound from a sound source is transformed when it is reflected off the portion of the pinna and determines a transfer function that characterizes the transformation of the sound. By determining a transfer function corresponding to reflections off the portion of the pinna, the system more accurately determines the effect of the pinna on the sound produced by the sound source. The system correlates the transfer function with an ear-side equalization filter that defines a transfer function from, for example, a portion of the pinna. Figure 1A How the sound of the sound source of the speaker 160 is perceived at the entrance of the user's ear canal. In fact, the sound represented by the ear equalization filter is the sound that would be perceived at the entrance of the user's ear canal if the auricle does not cause reflection of the sound. The system can use the ear equalization filter to adjust the audio content so that the adjusted audio content appears to arrive from the direction of the sound source after being reflected by the user's specific ear. In this way, the audio system minimizes the distortion of spatial cues in the audio content and provides spatialized audio content that is personalized for the user.

[0036] In some embodiments, the system determines the ear-side equalization filter that best corresponds to the transfer function of the reflection leaving the ear by referring to the ear-side equalization filter database. The database may include associations between acoustic transfer functions and ear-side equalization filters.

[0037] The system uses an acoustic sensor of a sensor array to capture sound and determines a transfer function corresponding to the transformation of sound at the ear canal caused by reflection from the user's auricle. The system correlates the transfer function with those stored in a database to determine an ear-side equalization filter that corresponds to or best corresponds to the transfer function. The acoustic properties of different users' ears may be different, resulting in different transfer functions and different ear-side equalization filters. In this way, transforming the audio content using the ear-side equalization filter preserves the individual spatial cues and individual equalization of the audio content.

[0038] In some embodiments, each of the ear equalization filters in the database can be generated by placing an inner ear acoustic sensor, such as an acoustic sensor at the entrance of the ear canal of the user's ear, capturing the sound from the sound source, and determining the transformation between the captured sound and the sound at the sound source. The inner ear acoustic sensor generates audio data indicating the perception of the sound at the entrance of the ear canal. Each of the ear equalization filters in the ear equalization filter can be associated with a set of transfer functions that determine how the auricle of the user's ear transforms the sound. Different directions of arrival can correspond to different ear equalization filters and transfer functions for each ear. The database can also store ear equalization filters and transfer functions corresponding to multiple individuals. In some embodiments, the database can include multiple ear equalization filters and transfer functions for a single individual.

[0039] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect on the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or otherwise used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0040] System Overview

[0041] Figure 1Ais a perspective view of a headset 100 implemented as an eyewear device according to one or more embodiments. In some embodiments, the eyewear device is a near eye display (NED). Typically, the headset 100 can be worn on the user's face so that content (e.g., media content) is presented using a display accessory and / or an audio system. However, the headset 100 can also be used so that media content is presented to the user in a different manner. Examples of media content presented by the headset 100 include one or more images, videos, audio, or some combination thereof. The headset 100 includes a frame and may include, among other components, a display accessory including one or more display elements 120, a depth camera accessory (DCA), an audio system, and a position sensor 190. Although Figure 1A The example locations on the headset 100 illustrate components of the headset 100, but these components may be located elsewhere on the headset 100, on a peripheral device that is paired with the headset 100, or some combination thereof. Similarly, there may be more than one component on the headset 100. Figure 1A More or fewer components than shown.

[0042] The frame 110 holds the other components of the headset 100. The frame 110 includes a front portion that holds the front portion of one or more display elements 120 and end pieces (e.g., temples) that attach to the user's head. The front portion of the frame 110 bridges the top of the user's nose. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that bends behind the user's ear (e.g., temple tip, ear piece).

[0043] One or more display elements 120 provide light to a user wearing the headset 100. As illustrated, the headset includes a display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light that is provided to the eyebox of the headset 100. The eyebox is the spatial location occupied by the user's eyes when the headset 100 is worn. For example, the display element 120 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is input coupled into one or more waveguides, and the light is output in such a way that a pupil replica exists in the eyebox of the headset 100. The input coupling (in-coupling) and / or output coupling (outcoupling) of light from one or more waveguides can be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a reflector, etc.) that scans the light from the light source when the light from the light source is input coupled into the one or more waveguides. Note that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a local area around the headset 100. The local area is an area surrounding the headset 100. For example, the local area may be a room where a user wearing the headset 100 is located, or the user wearing the headset 100 may be outdoors and the local area is an outdoor area. In this scenario, the headset 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent so that light from the local area can be combined with light from one or more display elements to produce AR and / or MR content.

[0044] In some embodiments, display element 120 does not generate image light, but is a lens that transmits light from a localized area to the eye frame. For example, one or both of display elements 120 can be a non-corrective lens (non-prescription) or a prescription lens (e.g., single vision, bifocal, and trifocal or progressive) to help correct the user's vision defects. In some embodiments, display element 120 can be polarized and / or tinted to protect the user's eyes from sunlight.

[0045] Note that in some embodiments, the display element 120 may include an additional optical block (not shown). The optical block may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that direct light from the display element 120 to the eye frame. The optical block may, for example, correct aberrations in some or all of the image content, magnify some or all of the images, or some combination of the foregoing.

[0046] The DCA determines depth information of a portion of a local area surrounding the headset 100. The DCA includes one or more imaging devices 130 and a DCA controller ( Figure 1A 140). In some embodiments, the illuminator 140 illuminates a portion of the local area with light. The light may be, for example, structured light in infrared (IR) (e.g., dot pattern, stripe, etc.), IR flash for time of flight, etc. In some embodiments, one or more imaging devices 130 capture an image of a portion of the local area including light from the illuminator 140. As illustrated, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, there is no illuminator 140 and there are at least two imaging devices 130.

[0047] The DCA controller uses the captured images and one or more depth determination techniques to calculate depth information for portions of the local area. The depth determination techniques may be, for example, direct time of flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from illuminator 140), some other technique for determining scene depth, or some combination of the foregoing.

[0048] The audio system provides spatialized audio content to a user. The audio system includes a transducer array, a sensor array, and an audio controller 150. However, in some other embodiments, the audio system may include different and / or additional components. Similarly, in some cases, the functionality described with reference to the components of the audio system may be distributed among the components in a manner different from that described herein. For example, some or all of the functions of the controller may be performed by a remote server.

[0049] The transducer array presents sound to the user. The transducer array includes multiple transducers. The transducer can be a speaker 160 or a tissue transducer 170 (e.g., a bone conduction transducer or a cartilage conduction transducer). The speaker 160 can be enclosed in the frame 110. In some embodiments, the headset 100 includes a speaker array that includes multiple speakers integrated into the frame 110 to improve the directionality of the presented audio content. In some embodiments, the speakers 160 can each be placed in the user's ear canal. The speakers 160 can be placed at other locations in the headset 100. The tissue transducer 170 is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to generate sound. The number and / or position of the transducers can vary depending on the user's ear. Figure 1A The difference shown in .

[0050] The sensor array detects sounds in a local area of ​​the headset 100. The sensor array includes a plurality of acoustic sensors 180. The acoustic sensors 180 capture sounds emitted from one or more sound sources in a local area (e.g., a room). Each acoustic sensor is configured to detect sounds and convert the detected sounds into an electronic format (analog or digital). The acoustic sensor 180 may be a sound wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sounds.

[0051] In some embodiments, one or more acoustic sensors 180 can be placed in the ear canal of each ear (e.g., acting as binaural microphones). In some embodiments, acoustic sensors 180 can be placed on an exterior surface of headset 100, on an interior surface of headset 100, separate from headset 100 (e.g., part of some other device), or some combination of the foregoing. The number and / or location of acoustic sensors 180 can vary depending on the device. Figure 1A For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected and the sensitivity and / or accuracy of the information. The acoustic detection locations can be oriented so that the microphone can detect sound in a wide range of directions around the user wearing the headset 100.

[0052] The audio controller 150 adjusts the audio content and instructs the transducer array to present the spatialized audio content to the user. The audio controller 150 adjusts the audio content according to the ear-side equalization filter that captures the response of the auricle of the user's ear to the audio signal. The audio controller 150 uses a beamformer to detect the reflection of the sound from a specific position of the auricle, and characterizes the transformation of the sound caused by the reflection as a transfer function. The transfer function is mapped to the ear-side equalization filter, which the audio controller 150 uses when rendering the spatialized audio content personalized for the user.

[0053] The audio controller 150 processes information from the sensor array that describes the sounds detected by the sensor array. The audio controller 150 may include a processor and a computer-readable storage medium. The audio controller 150 may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and / or a head-related transfer function), track the location of a sound source, form a beam in the direction of the sound source, classify the sound source, generate a sound filter for the speaker 160, or some combination of the foregoing.

[0054] The position sensor 190 generates one or more measurement signals in response to the movement of the headset 100. The position sensor 190 can be located on a portion of the frame 110 of the headset 100. The position sensor 190 can include an inertial measurement unit (IMU). Examples of the position sensor 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU, or some combination of the foregoing. The position sensor 190 can be located outside the IMU, inside the IMU, or some combination of the foregoing.

[0055] In some embodiments, the headset 100 can provide simultaneous localization and mapping (SLAM) for the position of the headset 100 and updates to the model of the local area. For example, the headset 100 may include a passive camera accessory (PCA) that generates color image data. The PCA may include one or more RGB cameras that capture images of some or all of the local areas. In some embodiments, some or all of the imaging devices 130 of the DCA may also be used as a PCA. The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area, update the model of the local area, or some combination of the foregoing. In addition, the position sensor 190 tracks the position (e.g., position and posture) of the headset 100 in the room. The following is combined with Figure 2-Figure 5 Additional details regarding the components of headset 100 are discussed.

[0056] Figure 1B 1 is a perspective view of a headset 105 implemented as an HMD according to one or more embodiments. In embodiments describing an AR system and / or an MR system, a portion of the front side of the HMD is at least partially transparent in the visible band (-380nm to 750nm), and a portion of the HMD between the front side of the HMD and the user's eyes is at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a band 175. The headset 105 includes many of the same components as described above with reference to FIG. Figure 1A The components described are the same components, but modified to be integrated with the HMD form factor. For example, the HMD includes a display assembly, a DCA, an audio system, and a position sensor 190. Figure 1B An illuminator 140 , a plurality of speakers 160 , a plurality of imaging devices 130 , a plurality of acoustic sensors 180 , and a position sensor 190 are shown.

[0057] Figure 22 is a cross-sectional diagram 200 of a user's ear illustrating reflection points on portions of the ear according to one or more embodiments. The ear includes an auricle 210, an ear canal 220, and an ear drum 230. A plurality of reflection points 240A-240F are located on various portions of the auricle.

[0058] A headset, such as headset 100 and / or headset 105, generates beamformers, each of which is configured to be directed toward a portion of the pinna 210 of the user's ear. A beamformer is a portion of an audio system that is configured to isolate an audio signal specific to a certain location. In some embodiments, a beamformer can isolate an audio signal specific to a sound source. Each of the beamformers can be directed toward a portion of the pinna 210 corresponding to each of the reflection points 240A-240F. A controller of the headset can generate the beamformers.

[0059] The transducer array of the headset or some other sound source produces sound that reflects off the user's pinna from reflection points 240A-240F. The reflected sound can be characterized by a transfer function associated with the location of each beamformed signal. The controller can determine from multiple transfer functions associated with reflections off the user's pinna how the sound may be perceived at the center of the user's ear relative to the location of the headset. The center of the user's ear may be the entrance to the ear canal 220. The controller can query a database of transfer functions associated with "ear-side" equalization filters to find an ear-side equalization filter that may be the best match for the user. The ear-side equalization filter characterizes how the sound is perceived at the entrance to the ear canal 220. The determination of the ear-side equalization filter will be related to Figure 3-Figure 4 Further discussion. The controller can adjust the audio content accordingly and present it to the user. For each ear, the different arrival directions of the sound can include different transfer functions and different ear-side equalization filters for each reflection point in reflection points 240. In some embodiments, the reflections leaving the user's auricle can result in different transfer functions and different ear-side equalization filters for each reflection point in reflection points 240.

[0060] Figure 3 is a block diagram of an example audio system 300 in accordance with one or more embodiments. Figure 1A or Figure 1B The audio system in may be an embodiment of the audio system 300. The audio system 300 provides personalized and spatialized audio content for a user by modifying the audio content using an ear-side equalization filter determined by capturing audio data using an acoustic sensor of a sensor array 320 located away from the user's ear. The sensors of the sensor array 320 capture portions of the reflections off the user's pinna using a beamformer directed toward each of the portions of the pinna (e.g., Figure 2The audio system 300 generates an acoustic transfer function corresponding to each of the reflection points and determines an ear equalization filter from the acoustic transfer function, which defines a sound transformation from the sound source to the center of the user's ear. Based on the ear equalization filter, the audio system 300 adjusts the audio content for the user's ear. The audio system 300 can perform a similar process for both ears to generate spatialized audio content that is personalized for the specific shape and other acoustic properties of the user's ear. Figure 3 In the embodiment of the present invention, the audio system 300 includes a transducer array 310, a sensor array 320, and an audio controller 330. Some embodiments of the audio system 300 have components different from those described here. Similarly, in some cases, the functions can be distributed among the components in a manner different from that described here.

[0061] The transducer array 310 is configured to present audio content. At least a portion of the sound generated by the transducer array 310 is received by the acoustic sensor in the sensor array 320. The transducer array 310 includes a plurality of transducers. A transducer is a device that provides audio content. The transducer may be, for example, a speaker (e.g., speaker 160), a tissue transducer (e.g., tissue transducer 170), some other device that provides audio content, or some combination of the foregoing. The tissue transducer may be configured to be used as a bone conduction transducer or a cartilage conduction transducer. The transducer array 310 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination of the foregoing. In some embodiments, the transducer array 310 may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range, and a moving coil transducer may be used to cover a second part of a frequency range.

[0062] The bone conduction transducer generates sound pressure waves by vibrating the bone / tissue in the user's head. The bone conduction transducer can be coupled to a portion of the headset and can be configured to be behind the pinna that is coupled to a portion of the user's skull. The bone conduction transducer receives vibration instructions from the audio controller 330 and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate tissue-loaded sound pressure waves that bypass the eardrum and propagate toward the user's cochlea.

[0063] The bone conduction transducer generates sound pressure waves by vibrating one or more portions of the ear cartilage of the user's ear. The bone conduction transducer can be coupled to a portion of the headset and can be configured to couple to one or more portions of the ear cartilage of the ear. For example, the bone conduction transducer can be coupled to the back of the auricle of the user's ear. The bone conduction transducer can be located anywhere along the ear cartilage around the outer ear (e.g., the auricle, the tragus, some other portion of the ear cartilage, or some combination of the foregoing). Vibrating one or more portions of the ear cartilage can generate: air-loaded sound pressure waves outside the ear canal; tissue-loaded sound pressure waves that vibrate certain portions of the ear canal to generate air-borne sound pressure waves within the ear canal; or some combination of the foregoing. The generated air-loaded sound pressure waves propagate along the ear canal toward the eardrum.

[0064] The transducer array 310 generates sound according to instructions from the audio controller 330. For example, the audio content can be a linear scan, a logarithmic scan, white noise, pink noise, a maximum length signal, an arbitrary signal, or some combination thereof. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that appears to originate from a specific direction and / or target area (e.g., objects and / or virtual objects in a local area). For example, spatialized audio content can make the sound appear to originate from a virtual singer across the room from the user of the audio system 300. The transducer array 310 can be coupled to a wearable device (e.g., headphones 100 or headphones 105). In an alternative embodiment, the transducer array 310 can be a plurality of speakers separated from the wearable device (e.g., coupled to an external console).

[0065] The sensor array 320 detects sound. The sound may come from a local area around the user of the headset, generated by the transducer array 310 of the headset, or some combination of the foregoing. The sensor array 320 may include a plurality of acoustic sensors, each of which detects the air pressure changes of the sound waves and converts the detected sound into acoustic content in an electronic format (analog or digital). The plurality of acoustic sensors may be placed on the headset (e.g., headset 100 and / or headset 105), on the user (e.g., in the ear canal of the user), on the neckband, or some combination thereof. In some embodiments, the acoustic sensors of the sensor array are located away from the user's ear canal. The acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, the sensor array 320 is configured to use at least some of the plurality of acoustic sensors to monitor the audio content generated by the transducer array 310. Increasing the number of sensors may improve the accuracy of the information (e.g., directionality) describing the sound field generated by the transducer array 310 and / or the sound from the local area.

[0066] The audio controller 330 controls the operation of the audio system 300. In particular, the audio controller 330 determines a transfer function that characterizes the response of the user's pinna to sound, and determines an ear equalization function that will help produce spatialized audio content. Figure 3 In some embodiments, the audio controller 330 includes a data repository 335, a DOA estimation module 340, a transfer function module 350, a tracking module 360, a beamforming module 370, and an equalization filter module 380. In some embodiments, the audio controller 330 can be located inside the headset. Some embodiments of the audio controller 330 have different components than those described here. Similarly, the functions can be distributed among the components in a manner different from that described here. For example, some functions of the controller can be performed outside the headset.

[0067] The data repository 335 stores data for use by the audio system 300. The data in the data repository 335 may include sounds recorded in a local area of ​​the audio system 300, audio content, head-related transfer functions (HRTFs), transfer functions for one or more sensors, array transfer functions (ATFs) for one or more acoustic sensors in an acoustic sensor, sound source locations, virtual models of local areas, direction of arrival estimates, sound filters, and other data related to use by the audio system 300, or any combination of the foregoing items. Once the ear-side equalization filter is determined, the data repository 335 may also store the ear-side equalization filter together with an associated set of transfer functions in a database of ear-side equalization filters. Each ear-side equalization filter in the stored ear-side equalization filters may be associated with the shape of the user's auricle, the user's position, a sound source, or a combination of the foregoing items. The data repository 335 may also store transfer functions that characterize the user's auricle's response to sound. In some embodiments, for each DOA estimate and each ear, the data repository 335 stores multiple transfer functions and ear-side equalization filters, each transfer function corresponding to a position on the user's auricle.

[0068] The DOA estimation module 340 is configured to locate sound sources in a local area based in part on information from the sensor array 320. Localization is the process of determining where a sound source is located relative to a user of the audio system 300. The DOA estimation module 340 performs a DOA analysis to locate one or more sound sources in a local area. The DOA analysis may include analyzing the intensity, spectrum, and / or arrival time of each sound at the sensor array 320 to determine the direction from which the sound originates. In some cases, the DOA analysis may include any suitable algorithm for analyzing the surrounding acoustic environment in which the audio system 300 is located.

[0069] For example, DOA analysis can be designed to receive an input signal from the sensor array 320 and apply a digital signal processing algorithm to the input signal to estimate the direction of arrival. These algorithms may include, for example, a delay and sum algorithm, in which the input signal is sampled, and the resulting weighted and delayed versions of the sampled signal are averaged together to determine the DOA. A least mean square (LMS) algorithm can also be implemented to create an adaptive filter. The adaptive filter can then be used to, for example, identify differences in signal strength, or differences in arrival time. These differences can then be used to estimate the DOA. In another embodiment, the DOA can be determined by transforming the input signal into the frequency domain and selecting a specific bin in the time-frequency (TF) domain for processing. Each selected TF interval can be processed to determine whether the bin includes a portion of the audio spectrum with a direct path audio signal. Those bins with a portion of the direct path signal can then be analyzed to identify the angle at which the sensor array 320 receives the direct path audio signal. The determined angle can then be used to identify the DOA for the received input signal. Other algorithms not listed above can also be used alone or in combination with the above algorithms to determine the DOA.

[0070] In some embodiments, the DOA estimation module 340 may also determine the DOA with respect to the absolute position of the audio system 300 within the local area. The position of the sensor array 320 may be received from an external system (e.g., some other component of the headset, an artificial reality console, a mapping server, a position sensor (e.g., the position sensor 190), etc.). The external system may create a virtual model of the local area, in which the local area and the position of the audio system 300 are mapped. The received position information may include the position and / or orientation of some or all of the audio system 300 (e.g., the sensor array 320). The DOA estimation module 340 may update the estimated DOA based on the received position information.

[0071] The transfer function module 350 is configured to generate one or more acoustic transfer functions. In general, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function module 350 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how the microphone receives the sound reflected off the user's auricle, i.e., the transformation of the sound caused by the reflection of the portion leaving the user's auricle.

[0072] ATF includes multiple transfer functions that characterize the relationship between the corresponding sound received by the acoustic sensor in the sound source and the sensor array 320. Therefore, for the sound source, there is a corresponding transfer function for each acoustic sensor in the sensor array 320. And this group of transfer functions is collectively referred to as ATF. Therefore, for each sound source, there is a corresponding ATF. Note that the sound source can be, for example, someone or something, a user, or one or more transducers in the transducer array 310 that generate sound in a local area. The ATF relative to the specific sound source position of the sensor array 320 can be different from user to user, because the anatomical structure of the person (e.g., ear shape, shoulder, etc.) can affect the sound when the sound propagates to the human ear. Therefore, the ATF of the sensor array 320 is personalized for each user of the audio system 300. The ATF of the sensor array 320 can be used to determine the sound pressure measurement at the center of the user's ear (such as at the entrance of the user's ear canal).

[0073] The transfer function module 350 can determine the ATF that characterizes the sound transformation by comparing the audio data generated by the acoustic sensors in the sensor array 320 with and without reflections from the ear. The transfer function module 350 instructs the transducer array 310 to present the sound when the user wears a headset. The beamformer discussed in further detail about the beamforming module enhances the sound reflected from the portion of the user's auricle. The acoustic sensors of the sensor array 320 generate audio data corresponding to the sound detected by the beamformer via the beamformed signal. The transfer function module 350 also instructs the transducer array 310 to present the sound when the user is not wearing a headset. The beamformer points to the same location, but because the user is not wearing a headset, the sound will not reflect away from the user's auricle. The sensor array generates audio data that captures the sound without reflections from the user's ear. The transfer function module 350 uses the beamformed signal to generate a calibration signal corresponding to the audio data detected without reflection. The transfer function module 350 determines the ATF by comparing the beamformed signal and the calibration signal. In some embodiments, calibration, i.e., capturing sound without reflections leaving the user's ears, can be performed in an anechoic chamber. In some embodiments, a head and / or torso simulator can be used to determine acoustic data that captures the sound of reflections leaving the user's pinna.

[0074] The tracking module 360 ​​is configured to track the position of one or more sound sources. The tracking module 360 ​​can compare current DOA estimates and compare them with a stored history of previous DOA estimates. In some embodiments, the audio system 300 can recalculate the DOA estimate periodically, such as once per second or once per millisecond. The tracking module can compare the current DOA estimate with the previous DOA estimate, and in response to a change in the DOA estimate of the sound source, the tracking module 360 ​​can determine that the sound source has moved. In some embodiments, the tracking module 360 ​​can detect changes in position based on visual information received from a headset or some other external source. The tracking module 360 ​​can track the movement of one or more sound sources over time. The tracking module 360 ​​can store the number of sound sources at each time point and the value of the position of each sound source. In response to changes in the value of the number or position of the sound source, the tracking module 360 ​​can determine that the sound source has moved. The tracking module 360 ​​can calculate an estimate of the positioning variance. The positioning variance can be used as a confidence level for each determination of the movement change.

[0075] The beamforming module 370 is configured to process one or more ATFs to selectively emphasize the sound from the sound source in a specific area while de-emphasize the sound from other areas. When analyzing the sound detected by the sensor array 320, the beamforming module 370 can combine information from different acoustic sensors to emphasize the sound associated with the specific area of ​​the local area while de-emphasize the sound from outside the area. The beamforming module 370 can isolate the audio signal associated with the sound from the specific sound source from other sound sources in the local area based on, for example, different DOA estimation and tracking modules 360 from the DOA estimation module 340. The beamforming module 370 can therefore selectively analyze discrete sound sources in the local area. In some embodiments, the beamforming module 370 can enhance the signal from the sound source. For example, the beamforming module 370 can apply a sound filter that eliminates signals above certain frequencies, below certain frequencies, or between certain frequencies. Signal enhancement is used to enhance the sound associated with a given identified sound source relative to other sounds detected by the sensor array 320.

[0076] The beamforming module 370 can generate beamformers, each beamformer being directed toward a portion of the user's auricle (e.g., reflection point 240). In some embodiments, the beamformers can be configured to sweep around the auricle or around the entire user's ear. The beamformed signal can enhance the sound reflected off the portion of the auricle that is detected by the acoustic sensors of the sensor array 320. The beamforming module 370 can generate the beamformers based on maximum directivity, minimum variance distortionless response, linear constrained minimum variance, or some combination of the foregoing.

[0077] The equalization filter module 380 determines the ear equalization filter and adjusts the audio content accordingly. The adjusted audio content can be the spatialized audio content customized for individual users. In one embodiment, the ear equalization filter specific to the user can be determined by placing the in-ear acoustic sensor at the entrance of the ear canal of the user's ear (i.e. the center of the ear). The in-ear acoustic sensor can be a part of the sensor array 320. The audio data generated by the in-ear acoustic sensor can be used to determine the transformation of the response characterizing the center of the ear relative to the sound at the sound source. The ear equalization filter can be stored in the data repository 335 in the database of the ear equalization filter. Each ear equalization filter in the ear equalization filter corresponds to a group of transfer functions, and the group of transfer functions characterizes how the user's auricle transforms sound. The database of ear equalization filter and transfer function is determined from multiple users. In some embodiments, a single user can have multiple ear equalization filters and associated transfer functions stored in the database.

[0078] In some embodiments, the ear equalization filter for the user's ear can be determined by referring to the database of the ear equalization filter stored in the data repository 335. The transfer function module 350 can determine the ATF of the sound transformation at each reflection point of the characterizing auricle, wherein the equalization filter module 380 subsequently relates the ATF to the reference ear equalization filter stored in the database. The transfer function associated with the ear equalization filter can exactly and / or closely match the ATF. The ear equalization filter can change based on the type of sound received by the user's ear, the shape of the user's auricle, the position of the user or some combination of the aforementioned items. The database of the ear equalization filter is referenced to eliminate the need for in-ear acoustic sensors. Instead, the response of the center of the user's ear can be detected remotely by detecting the transformation of the sound of the user's auricle that reflects away from the center of the user's ear. By using the trained neural network that uses the ATF as input and outputs the appropriate ear equalization filter, the ear equalization filter that matches closely can be found automatically.

[0079] In some embodiments, the ear equalization filter spatializes the audio content so that the audio content appears to originate from the target area or arrival direction. The equalization filter module 380 can use HRTF and / or acoustic parameters to generate sound filters. Acoustic parameters describe the acoustic properties of the local area. Acoustic parameters can include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the equalization filter module 380 calculates one or more of the acoustic parameters. In some embodiments, the equalization filter module 380 requests acoustic parameters from a mapping server (e.g., as described below with respect to Figure 5 described).

[0080] The equalization filter module 380 can provide the spatialized audio content generated using the ear-side equalization filter to the transducer array 310, and the transducer array 310 presents the spatialized audio content to the user in response. The spatialized audio content may include different audio content for the left ear and the right ear, thereby providing spatial cues.

[0081] Figure 4 4 is a flow diagram of a process 400 for generating spatialized audio content personalized for a user's ears in accordance with one or more embodiments. The process may be performed by an audio system (e.g., audio system 300) coupled to headphones (e.g., headphones 100 and / or headphones 105). In other embodiments other entities may perform some or all of the steps of the process (e.g., a console). Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0082] The audio system generates 410 audio data using an acoustic sensor of the sensor array. For example, the acoustic sensor generates audio data by transforming one or more sounds into electrical signals. The one or more sounds may be generated by a sound source and arrive at the acoustic sensor from a specific direction of arrival. The one or more sounds may be generated by the audio system (e.g., one or more transducers of the transducer array 310) or may be generated by one or more sound sources separate from the audio system.

[0083] The audio system generates 420 a beamformed signal by processing audio data using a beamformer for an acoustic sensor of a sensor array. Each beamformer in the beamformer is directed to a different portion of the pinna of the user's ear so that the beamformed signal corresponds to a reflection of the sound from the portion of the pinna. The beamformed signal can be generated from one or more sounds. For example, the audio system can generate a sound, in response to which the audio system can apply each beamformer in the beamformer. In another embodiment, the audio system can generate multiple sounds, wherein a beamformer is applied to each sound to systematically cover different portions of the pinna. The beamformer can sequentially and systematically cover different portions of the pinna so that the beamformer sweeps across the user's ear. For example, the audio system can generate a first sound, in response to which the acoustic sensor of the sensor array can generate corresponding first audio data. The first beamformer can be directed to a first portion of the ear, and the audio system can use the first beamformer to generate a first beamformed signal when processing the first audio data. This process can be repeated for multiple sounds and beamformers until beamformed signals from most of the pinna are covered. The beamformed signals can collectively indicate a sound pressure measurement at the center of the user's ear.

[0084] In some embodiments, the sound produced by the audio system can be presented to the user via tissue conduction. In this case, the beamformed signal corresponds to the transformation of the sound caused by the vibration of different parts of the auricle.

[0085] The audio system determines 430 a transfer function using the beamformed signal. The transfer function defines the transformation of the sound caused by the reflection of different parts of the auricle of the user's ear. Each part of the auricle and the beamformed signal can correspond to a different transfer function. In some embodiments, the transfer function can be determined by comparing the beamformed signal and a calibration signal, which defines the sound from the sound source without the reflection of the part of the ear auricle. The audio system can generate a calibration signal in which the same beamformer is used when the user does not wear a headset. The audio system can process the audio data generated by the acoustic sensor in this way to determine the calibration signal. The transfer function is used to generate spatialized audio content for the ear, as discussed in more detail below. In some embodiments, the audio system can generate the sound of the part of the auricle that reflects off the ear. The reflected acoustics of the sound leaving the auricle are processed by the acoustic sensor to generate audio data. By deconvolving the audio data corresponding to the reflection of the part of the ear with the sound generated by the audio system, the transfer function for the reflection of each part leaving the auricle can be determined.

[0086] The audio system determines 440 an ear equalization filter for the ear based on the transfer function. The ear equalization filter defines a transformation of the sound at the center of the user's ear (e.g., ear canal) that is personalized for the user. In some embodiments, the audio system can use the transfer function to find a database of reference ear equalization filters and determine a matching or best matching ear equalization filter for the determined transfer function. Each reference ear equalization filter stored in the database of reference ear equalization filters can be associated with a different set of transfer functions.

[0087] The audio system can determine a set of transfer functions stored in a database by using at least one acoustic sensor of a sensor array placed in the ear of the user. The acoustic sensor can be placed at the entrance of the ear canal of the ear of the user. A sound source generates one or more sounds. The auricle of the user's ear reflects the sound. The acoustic sensor at the entrance of the ear canal of the user produces audio data that captures how the sound is perceived at the center of the user's ear, while the acoustic sensor of the sensor array far from the ear captures the reflection of the sound leaving the auricle. The audio system determines a set of transfer functions that characterize the transformation of the sound caused by the reflection leaving the auricle. The audio system correlates the transfer function with the response at the center of the ear to determine an ear-side equalization filter for the set of transfer functions. The audio system stores the ear-side equalization filter and the associated transfer function in a database for future reference.

[0088] The audio system generates 450 spatialized audio content for the ear using the ear-side equalization filter. In addition, the audio system can present the spatialized audio content to the ear, for example, to a transducer located at the ear. Process 400 can be repeated for the other ear of the user. In one example, process 400 is performed in parallel for the left ear and the right ear to generate spatialized audio content for both ears. Different ears can include different beamformed signals and transfer functions, thereby generating different ear-side equalization filters for each ear.

[0089] Figure 5 is a block diagram of an example artificial reality system 500 according to one or more embodiments. According to one or more embodiments, the system 500 includes a headset 505. In some embodiments, the headset 505 may be Figure 1A Headphones 100 or Figure 1B The system 500 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 5 The illustrated system 500 includes a headset 505, an input / output (I / O) interface 510 coupled to a console 515, a network 520, and a mapping server 525. Figure 5 An example system 500 is shown that includes one headset 505 and one I / O interface 510, but in other embodiments, any number of these components may be included in the system 500. For example, there may be multiple headsets, each with an associated I / O interface 510, each headset and I / O interface 510 communicating with a console 515. In alternative configurations, different and / or additional components may be included in the system 500. Additionally, in some embodiments, the system 500 may include a plurality of headsets, each with an associated I / O interface 510, each headset and I / O interface 510 communicating with a console 515. Figure 5The functionality described in one or more of the components shown in the figure may be combined with Figure 5 The described methods may be distributed among the components in different ways. For example, some or all of the functionality of the console 515 may be provided by the headset 505.

[0090] Headphones 505 include a display assembly 530, an optical block 535, one or more position sensors 540, and a DCA 545. Some embodiments of headphones 505 have a combination of Figure 5 In addition, in some other embodiments, by combining Figure 5 The functionality provided by the various components described may be variously distributed among the components of headset 505 , or captured in a separate accessory remote from headset 505 .

[0091] Display accessory 530 displays content to the user based on data received from console 515. Display accessory 530 displays content using one or more display elements (e.g., display element 120). The display element can be, for example, an electronic display. In various embodiments, display accessory 530 includes a single display element or multiple display elements (e.g., a display for each eye of the user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination of the foregoing. Note that in some embodiments, display element 120 can also include some or all of the functionality of optical block 535.

[0092] The optical block 535 can amplify image light received from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or both of the eye boxes of the headset 505. In various embodiments, the optical block 535 includes one or more optical elements. Example optical elements included in the optical block 535 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 535 can include a combination of different optical elements. In some embodiments, one or more of the optical elements in the optical block 535 can have one or more coatings, such as a partially reflective or anti-reflective coating.

[0093] The magnification and focusing of the image light by the optical block 535 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. In addition, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the display content is presented using almost all of the user's field of view (e.g., approximately 110 degrees diagonal) and in some cases all of the user's field of view. In addition, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0094] In some embodiments, the optical block 535 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberration, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optical block 535 corrects the distortion when it receives image light generated based on the content from the electronic display.

[0095] The position sensor 540 is an electronic device that generates data indicating the position of the headset 505. The position sensor 540 generates one or more measurement signals in response to the movement of the headset 505. The position sensor 190 is an embodiment of the position sensor 540. Examples of the position sensor 540 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, or some combination of the foregoing. The position sensor 540 may include multiple accelerometers that measure translational movement (forward / backward, up / down, left / right) and multiple gyroscopes that measure rotational movement (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the headset 505 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometer over time to estimate a velocity vector, and integrates the velocity vector over time to determine the estimated position of a reference point on the headset 505. The reference point is a point that can be used to describe the position of the headset 505. Although the reference point may generally be defined as a point in space, in practice the reference point is defined as a point within the headset 505 .

[0096] The DCA 545 generates depth information for a portion of a local area. The DCA includes one or more imaging devices and a DCA controller. The DCA 545 may also include an illuminator. The operation and structure of the DCA 545 are described above with respect to Figure 1A Described.

[0097] The audio system 550 provides spatialized audio content to the user of the headset 505. The audio system 550 is substantially the same as the audio system 300 described above. The audio system 550 may include one or more acoustic sensors, one or more transducers, and an audio controller. The audio system 550 may provide spatialized audio content to the user by inferring the response of the audio content at the center of the user's ear using audio data captured by an acoustic sensor array located away from the user's ear. The audio system 550 may determine a transfer function based on reflections of sound leaving the user's pinna, relate the transfer function to an ear-side equalization filter, and correspondingly generate spatial audio content presented to the user.

[0098] In some embodiments, the audio system 550 may request acoustic parameters from the mapping server 525 via the network 520. The acoustic parameters describe one or more acoustic properties of the local area (e.g., room impulse response, reverberation time, reverberation level, etc.). The audio system 550 may provide information describing at least a portion of the local area from, for example, the DCA 545 and / or position information of the headset 505 from the position sensor 540. The audio system 550 may generate one or more sound filters using one or more of the acoustic parameters received from the mapping server 525, and use the sound filters to provide audio content to the user.

[0099] The I / O interface 510 is a device that allows a user to send an action request and receive a response from the console 515. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end the capture of image or video data, or an instruction to perform a specific action within an application. The I / O interface 510 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 515. The action request received by the I / O interface 510 is transmitted to the console 515, and the console 515 performs an action corresponding to the action request. In some embodiments, the I / O interface 510 includes an IMU that captures calibration data, which indicates an estimated position of the I / O interface 510 relative to the initial position of the I / O interface 510. In some embodiments, the I / O interface 510 may provide tactile feedback to the user according to the instruction received from the console 515. For example, tactile feedback is provided when an action request is received, or the console 515 transmits instructions to the I / O interface 510 to cause the I / O interface 510 to generate tactile feedback when the console 515 performs an action.

[0100] The console 515 provides content to the headset 505 for processing based on information received from one or more of: the DCA 545, the headset 505, and the I / O interface 510. Figure 5 In the example shown in , the console 515 includes an application repository 555, a tracking module 560, and an engine 565. Some embodiments of the console 515 have Figure 5 Similarly, the functionality described further below may be combined with Figure 5 The described methods are distributed among the components of the console 515 in different ways. In some embodiments, the functionality discussed herein with respect to the console 515 can be implemented in the headset 505 or in a remote system.

[0101] The application repository 555 stores one or more applications for execution by the console 515. An application is a group of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by an application may be input received from a user in response to movement via the headset 505 or the I / O interface 510. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0102] The tracking module 560 tracks the movement of the headset 505 or the I / O interface 510 using information from the DCA 545, one or more position sensors 540, or some combination of the foregoing. For example, the tracking module 560 determines the location of a reference point of the headset 505 in a map of the local area based on the information from the headset 505. The tracking module 560 may also determine the location of an object or a virtual object. Additionally, in some embodiments, the tracking module 560 may use portions of the data from the position sensor 540 indicating the location of the headset 505 and the representation of the local area from the DCA 545 to predict the future location of the headset 505. The tracking module 560 provides the estimated or predicted future location of the headset 505 or the I / O interface 510 to the engine 565.

[0103] The engine 565 executes the application and receives the position information, acceleration information, speed information, predicted future position or some combination thereof of the headset 505 from the tracking module 560. Based on the received information, the engine 565 determines the content to be provided to the headset 505 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 565 generates content for the headset 505, which reflects the movement of the user in the virtual local area or in the local area that enhances the local area with additional content. In addition, the engine 565 performs an action in the application executed on the console 515 in response to the action request received from the I / O interface 510, and provides feedback to the user that the action has been performed. The feedback provided can be visual or auditory feedback via the headset 505 or tactile feedback via the I / O interface 510.

[0104] The network 520 couples the headset 505 and / or the console 515 to the mapping server 525. The network 520 may include any combination of local area networks and / or wide area networks using both wireless and / or wired communication systems. For example, the network 520 may include the Internet and a mobile phone network. In one embodiment, the network 520 uses standard communication technologies and / or protocols. Therefore, the network 520 may include the use of protocols such as Ethernet, 802.11, World Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the networking protocols used on the network 520 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. The data exchanged over the network 520 may be represented using formats and / or techniques including image data in binary form (e.g., portable network graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or some of the links may be encrypted using conventional encryption techniques such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.

[0105] The mapping server 525 may include a database storing virtual models describing multiple spaces, wherein a location in the virtual model corresponds to the current configuration of the local area of ​​the headset 505. The mapping server 525 receives information describing at least a portion of the local area and / or location information of the local area from the headset 505 via the network 520. The mapping server 525 determines a location in the virtual model associated with the local area of ​​the headset 505 based on the received information and / or location information. The mapping server 525 determines (e.g., retrieves) one or more acoustic parameters associated with the local area based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The mapping server 525 can transmit the location of the local area and any values ​​of the acoustic parameters associated with the local area to the headset 505.

[0106] Additional configuration information

[0107] The above description of the embodiments of the present disclosure is for illustrative purposes; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art can appreciate that many modifications and variations are possible based on the above disclosure.

[0108] Some parts of this specification sheet describe embodiments of the present disclosure based on algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are often used by technicians in the field of data processing to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. related to manufacturing processes. In addition, without loss of generality, it is also proven that it is convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination of the foregoing.

[0109] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described (e.g., with respect to a manufacturing process).

[0110] Embodiments of the present disclosure may also relate to a device for performing the operations herein. The device may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transient tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in this specification may include a single processor or may be an architecture that uses a multi-processor design to improve computing power.

[0111] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and may not be selected to define or limit the subject matter of the present invention. Therefore, the scope of the present disclosure is not intended to be limited by this specific embodiment, but by any claims issued based on the application herein. Therefore, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure set forth in the appended claims.

Claims

1. A method for an audio system, comprising: generating, by an acoustic sensor of the sensor array, audio data from one or more sounds received by the acoustic sensor; generating a beamformed signal by processing the audio data using beamformers, each beamformer being directed toward a respective portion of an ear of a user; defining a transfer function using a transformation of the sound caused by reflection of the beamformed signal from the portion of the ear; as well as Spatialized audio content is generated for the ear based on the transfer function.

2. The method of claim 1 , wherein generating the spatialized audio content based on the transfer function comprises: determining an ear-side equalization filter based on the transfer function; as well as The audio content is adjusted for the user using the ear-side equalization filter. 3 . The method of claim 2 , wherein determining the ear side equalization filter comprises referencing a database of reference ear side equalization filters. 4 . The method of claim 2 , wherein determining the ear-side equalization filter comprises correlating the transfer function with a filter calibrated for the user.

5. The method of claim 1 , wherein using the beamformed signal to determine the transfer function defining a transformation of the sound caused by reflection from the portion of the ear comprises: generating, by the acoustic sensor of the sensor array, other audio data from one or more other sounds received by the acoustic sensor in the absence of reflections from the portion of the ear; generating a calibration signal by processing the other audio data using the beamformer; as well as The transfer function is determined using the beamformed signal and the calibration signal.

6. The method according to claim 1, wherein: At least one acoustic sensor in the sensor array is positioned at the entrance of the ear canal of the ear of the user; as well as Using the beamformed signal to determine the transfer function defining a transformation of the sound caused by reflection from the part of the ear comprises: generating, by the at least one acoustic sensor of the sensor array, other audio data from one or more other sounds received by the at least one acoustic sensor; as well as The transfer function is determined using the beamformed signal and the other audio data.

7. The method of claim 5, wherein the beamformed signals collectively indicate a sound pressure measurement at a center of the ear of the user.

8. The method of claim 1, further comprising generating, by at least one transducer, the one or more sounds received by the acoustic sensor.

9. The method of claim 1, wherein each beamformer is directed toward a different portion of the pinna of the ear.

10. The method according to claim 1, further comprising: generating a first beamformer of the beamformers directed toward a first portion of the ear; generating, by the acoustic sensor of the sensor array, first audio data of the audio content from a first sound of the one or more sounds; as well as The first audio data is processed using the first beamformer to generate a first one of the beamformed signals.

11. An audio system comprising: A sensor array comprising an acoustic sensor and an audio controller, the system being configured to perform the method of claim 1, or wherein the acoustic sensor is configured to: generating audio data from one or more sounds received by the acoustic sensor; and The audio controller is configured to: generating a beamformed signal by processing the audio data using beamformers for the acoustic sensors of the sensor array, each beamformer directed toward a respective portion of an ear of the user; defining a transfer function using a transformation of the sound caused by reflection of the beamformed signal from the portion of the ear; as well as Spatialized audio content is generated for the ear based on the transfer function.

12. A computer-readable non-transitory storage medium storing instructions for presenting spatialized audio content, the instructions, when executed by a processor, cause the processor to perform the method according to claim 1 or to perform the steps comprising: generating, by an acoustic sensor of the sensor array, audio data from one or more sounds received by the acoustic sensor; generating a beamformed signal by processing the audio data using beamformers, each beamformer being directed toward a respective portion of an ear of a user; defining a transfer function using a transformation of the sound caused by reflection of the beamformed signal from the portion of the ear; as well as Spatialized audio content is generated for the ear based on the transfer function.

13. The computer-readable non-transitory storage medium of claim 12, wherein the instructions further cause the processor to perform steps comprising: determining an ear-side equalization filter based on the transfer function; and The audio content is adjusted for the user using the ear-side equalization filter.

Images