Methods, systems, and non-transitory computer-readable media for presenting audio content

Abstract

A system for presenting audio content to a user. The system includes one or more microphones (210) coupled to a frame (110) of a head-mounted viewer (100). The one or more microphones (210) capture sound from a local area. The system also includes an audio controller (150) integrated into the head-mounted viewer (100) and communicatively coupled to an in-ear device (185) worn by the user. The audio controller (150) identifies one or more sound sources in the local area based on the captured sound. The audio controller (150) also determines a target sound source of the one or more sound sources and determines one or more filters to apply to sound signals of the captured sound associated with the target sound source. The audio controller (150) also generates an enhanced sound signal by applying the one or more filters to the sound signals and provides the enhanced sound signal to the in-ear device (185) for presentation to the user.

Description

[0001] This application is a divisional application of the invention application filed on June 5, 2021, with application number 2021800603276, entitled "Method, System and Non-transitory Computer-readable Medium for Presenting Audio Content". Technical Field

[0002] This disclosure generally relates to generating audio content, and more specifically, to an audio system that uses multiple sound profiles that are personalized for each user. Background Technology

[0003] In environments with multiple sound sources, listeners may have difficulty focusing on a specific sound source while ignoring sounds from others. For example, in a noisy room where many people are talking simultaneously, a listener may struggle to distinguish a particular speaker's voice from the voices of others in the room. This phenomenon is known as the cocktail party problem. The cocktail party problem can cause even more serious problems for people with partial hearing loss. Hearing loss can be attributed to factors such as aging, trauma, and genetic defects. People with hearing loss are more severely affected when there are sound sources that interfere with the sound source of personal interest. Summary of the Invention

[0004] An audio system on a head-mounted viewer generates one or more filters applied to audio content before it is presented to a user. These filters can be generated based on the user's voice profile. The user's voice profile is personalized voice information describing their hearing at different frequencies. In some cases, the user may have some degree of hearing loss, which is captured in their voice profile. To compensate for some or all of the user's hearing loss, the sound signal in the audio content emitted by the sound source (i.e., the target sound source) can be amplified using one or more filters to generate an enhanced sound signal. In some embodiments, the head-mounted viewer presents the enhanced sound signal to the user (e.g., via a speaker assembly on the head-mounted viewer). In some embodiments, the head-mounted viewer provides the enhanced sound signal to one or more in-ear devices communicatively coupled to the head-mounted viewer, and these devices present the enhanced sound signal to the user. In some embodiments, the head-mounted viewer provides one or more filters to one or more in-ear devices, which apply the filters to the sound signal to generate the enhanced sound signal, and then present the enhanced sound signal to the user.

[0005] In some embodiments, one or more filters may be generated to suppress sound. For example, the one or more filters may be used to suppress audio content from one or more sound sources (e.g., a fan, a person who is not the target sound source, etc.) in a local area. And in some embodiments, the one or more filters may be used to suppress leakage signals from the in-ear device (enhanced sound signals leaking into the local area).

[0006] In some embodiments, a method for enhancing an audio signal for presentation to a user is described. The method includes acquiring sound from a local area using one or more microphones (e.g., one or more microphones arranged on a head-mounted viewer). The method further includes identifying one or more sound sources within the local area based on the acquired sound. The method further includes determining a target sound source among the one or more sound sources. The method further includes determining one or more filters to apply to an audio signal in the acquired sound associated with the target sound source. The method further includes generating an enhanced audio signal by applying the one or more filters to the audio signal. The method further includes providing the enhanced audio signal to a speaker assembly for presentation to a user. In some embodiments, the speaker assembly is located on a head-mounted viewer. In other embodiments, the speaker assembly is located on one or more in-ear devices.

[0007] In some embodiments, a system for enhancing an audio signal for presentation to a user is described. The system includes one or more microphones coupled to the frame of a head-mounted viewer. The microphones are configured to acquire sound from a localized region. The system also includes an audio controller integrated into the head-mounted viewer and communicatively coupled to an in-ear device worn by a user. The audio controller is configured to identify one or more sound sources in the localized region based on the acquired sound. The audio controller is further configured to determine a target sound source among the one or more sound sources. The audio controller is also configured to determine one or more filters to apply to the acquired sound signal associated with the target sound source. The audio controller is further configured to generate an enhanced audio signal by applying the one or more filters to the sound signal. The audio controller is also configured to provide the enhanced audio signal to the in-ear device for presentation to the user. Embodiments of a non-transitory computer-readable storage medium are also described, configured to store a plurality of instructions for performing the methods of this disclosure.

[0008] According to a first aspect of this disclosure, a method is provided, the method comprising: Sound is collected from a local area using one or more microphones; Based on the collected sound, identify one or more sound sources in a local area; Identify the target sound source among the one or more sound sources; Determine one or more filters to apply to the sound signals associated with the target sound source in the acquired sound; By applying one or more filters to the audio signal, an enhanced audio signal is generated; and Provides enhanced sound signals to the speaker components for presentation to the user.

[0009] The method may also include: The target sound source was detected to be within a threshold distance of a different sound source in a local region; and At least one of the one or more filters is updated such that the spatial interval between the target sound source and the different sound source is increased to at least a distance greater than a threshold distance. The enhanced sound signal presented to the user makes the sound sound like it originates from the target sound source, and the sound from the different sound source sounds like it originates from a location in the local area that is at least a distance away from the target sound source.

[0010] The one or more filters may be partially based on a user's voice profile, which includes at least a user's audiogram, and the method further includes: Based on user feedback, the user's voice profile has been updated.

[0011] The one or more filters may be based in part on one or more of the following: the user's location in the local area, the location of each of the one or more sound sources in the local area, one or more acoustic parameters of the local area, and various types of the multiple sound sources in the local area.

[0012] Determining one or more filters to apply to the sound signals associated with the target sound source in the acquired sound can include: Select a wide dynamic range compression filter corresponding to the target sound source from a plurality of wide dynamic range compression filters, wherein each wide dynamic range compression filter in the plurality of wide dynamic range compression filters corresponds to a different corresponding sound source among the plurality of sound sources.

[0013] Providing enhanced sound signals to the speaker components to be presented to the user includes: Provides enhanced sound signals to the speaker components in in-ear devices worn by users.

[0014] Collecting sound from a local area using one or more microphones can include: Sound is captured from a localized area by one or more microphones coupled to the frame of a head-mounted viewer, which is separate from and communicatively coupled to an in-ear device.

[0015] The method may also include: Leakage signals from in-ear devices are collected using one or more microphones. Determine a suppression signal for substantially preventing leakage, the suppression signal being out of phase with the leakage signal; and The suppression signal is provided as an output signal to a separate speaker assembly coupled to the head-mounted viewer.

[0016] The target sound source can be determined partly based on the user's line of sight.

[0017] The target sound source can be determined in part based on user-provided language prompts.

[0018] Determining the target sound source may include: Capture images of gestures made by the user; Determine that the posture corresponds to the selected sound source; and select the sound source as the target sound source.

[0019] This method can be implemented by a computer.

[0020] According to a second aspect of this disclosure, a system is provided, the system comprising: One or more microphones coupled to the frame of the head-mounted viewer, the one or more microphones being configured to collect sound from a local area; An audio controller, integrated into the head-mounted viewer and communicatively coupled to an in-ear device worn by the user, is configured to: Based on the collected sound, identify one or more sound sources in a local area; Identify the target sound source among the one or more sound sources; Determine one or more filters to apply to the sound signals associated with the target sound source in the acquired sound; By applying one or more filters to the audio signal, an enhanced audio signal is generated; and Provides enhanced sound signals to in-ear devices for presentation to users.

[0021] The audio controller can also be configured as follows: The target sound source was detected to be within a threshold distance of a different sound source in a local region; and At least one of the one or more filters is updated such that the spatial spacing between the target sound source and the different sound source is increased to at least a spacing distance greater than a threshold distance. The enhanced sound signal presented to the user makes the sound sound like it originates from the target sound source, and the sound from the different sound source sounds like it originates from a location in the local area that is at least a distance away from the target sound source.

[0022] Determining one or more filters to apply to the acquired sound signal associated with the target sound source may include: selecting a wide dynamic range compression filter corresponding to the target sound source from a plurality of wide dynamic range compression filters, wherein each of the plurality of wide dynamic range compression filters corresponds to a different corresponding sound source among the plurality of sound sources.

[0023] The one or more microphones can collect leakage signals from in-ear devices, and the audio controller can also be configured to: Determine a suppression signal for substantially preventing leakage, the suppression signal being out of phase with the leakage signal; and The suppression signal is provided as an output signal to a separate speaker assembly coupled to the head-mounted viewer.

[0024] The one or more filters may be based in part on one or more of the following: the user's location in the local area, the location of each of the one or more sound sources in the local area, one or more acoustic parameters of the local area, and various types of the multiple sound sources in the local area.

[0025] The target sound source can be determined by the audio controller based on the user's line of sight.

[0026] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing a plurality of instructions, which, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the operations including: Sound is collected from a local area using one or more microphones; Based on the collected sound, identify one or more sound sources in a local area; Identify the target sound source among the one or more sound sources; Determine one or more filters to apply to the sound signals associated with the target sound source in the acquired sound; By applying one or more filters to the audio signal, an enhanced audio signal is generated; and Provides enhanced sound signals to the speaker components for presentation to the user.

[0027] These instructions can also cause one or more processors to perform multiple operations, including: The target sound source was detected to be within a threshold distance of a different sound source in a local region; and At least one of the one or more filters is updated such that the spatial interval between the target sound source and the different sound source is increased to at least a distance greater than a threshold distance. The enhanced sound signal presented to the user makes the sound sound like it originates from the target sound source, and the sound from the different sound source sounds like it originates from a location in the local area that is at least a distance away from the target sound source.

[0028] Determining one or more filters to apply to the sound signals associated with the target sound source in the acquired sound can include: Select a wide dynamic range compression filter corresponding to the target sound source from a plurality of wide dynamic range compression filters, wherein each wide dynamic range compression filter in the plurality of wide dynamic range compression filters corresponds to a different corresponding sound source among the plurality of sound sources. Attached Figure Description

[0029] Figure 1A A perspective view of a headset implemented as an eyewear device according to one or more embodiments.

[0030] Figure 1B A perspective view of a head-mounted viewer implemented as a head-mounted display according to one or more embodiments.

[0031] Figure 2 This is a block diagram of an audio system according to one or more embodiments.

[0032] Figure 3 This is an example of a local region comprising one or more sound sources according to one or more embodiments.

[0033] Figure 4 A flowchart illustrating a process for enhancing an audio signal for presentation to a user, according to one or more embodiments, is provided.

[0034] Figure 5 A system comprising a head-mounted viewer and one or more in-ear devices according to one or more embodiments.

[0035] These accompanying drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles described herein. Detailed Implementation

[0036] An audio system on a head-mounted viewer generates one or more filters to apply to the audio content before it is presented to the user. The audio content may include replaying captured sound (from a local area surrounding the head-mounted viewer), virtual sound provided by the head-mounted viewer, music and / or audio (e.g., from a call) provided by a mobile device (e.g., a mobile phone), or some combination thereof. Sound includes one or more sound signals, each associated with a sound source. The sound source may be a real-world sound source, such as a user, a person, a person wearing the head-mounted viewer, an animal, a speaker (e.g., a speaker in an in-ear device), a mechanical device, or any other real-world object emitting sound in a local area surrounding the head-mounted viewer, a virtual sound source (e.g., a virtual object presented to the user in the local area), or a mobile device sound source (e.g., a mobile phone speaker, a tablet speaker, or any other mobile device speaker emitting sound in the local area). In some embodiments, the audio system generates one or more filters (e.g., low-pass filters, high-pass filters, dynamic range compression filters, etc.) to apply to the sound signal emitted by a specific sound source (i.e., a target sound source). The one or more filters applied to the audio signal may, for example, amplify certain frequencies of the audio signal (e.g., frequencies greater than 10 kHz), apply a specific gain to the audio signal (using a wide dynamic range compression filter), spatialize the audio signal, attenuate certain frequencies of the audio signal, equalize the intensity of certain frequencies, apply some other adjustment to the audio signal, apply a multi-band limiter to certain frequencies, or some combination thereof. In some embodiments, the audio system generates one or more filters to apply to one or more other audio signals emitted by one or more other sound sources. The one or more filters applied to the one or more other audio signals may reduce or eliminate specific frequencies of those audio signals. Any or all filters can be dynamically adjusted as the audio content changes over time.

[0037] An audio system can identify target sound sources in part based on implicit user feedback (e.g., eye-tracking information) stored in a model of a local region. This model tracks the position, motion, and sound of each sound source (including the user) within the local region and stores acoustic parameters describing that region. In some embodiments, the audio system can identify target sound sources based on explicit user input.

[0038] One or more filters generated by the audio system can be based on a user's voice profile. A user's voice profile is personalized voice information describing their hearing response to sounds at different frequencies. The voice profile may include audiograms, loudness discomfort level test results, speech-in-noise test results, spectro-temporal discrimination results, temporal fine-structure test results, cognitive performance test results, self-reported hearing ability, other hearing test results, or a combination thereof. In some cases, users may have some degree of hearing loss, which is captured in their voice profile. To compensate for some or all of a user's hearing loss, the sound signal emitted from a target sound source can be amplified using one or more filters to generate an enhanced sound signal. In some embodiments, to present the user with better sound quality, the sound signal emitted from the target sound source can be amplified to better spatialize the sound signal, or the sound signal emitted from the target sound source can be amplified to reduce feedback. In some embodiments, one or more additional sound signals emitted by other sound sources in the audio content can be suppressed using one or more filters. In some embodiments, the one or more filters generated by the audio system may be based on a model of a local region.

[0039] In some embodiments, an audio system may apply one or more filters to a sound signal emitted by an identified target sound source. In another embodiment, an in-ear device applies one or more filters to a sound signal. By applying these filters to the sound signal, an enhanced sound signal is generated. In some embodiments, a head-mounted viewer presents the enhanced sound signal to a user (e.g., via a speaker assembly on the head-mounted viewer). In some embodiments, the head-mounted viewer provides the enhanced sound signal to one or more in-ear devices (the in-ear devices are communicatively coupled to the head-mounted viewer), and the one or more in-ear devices present the enhanced sound signal to a user. In some embodiments, the head-mounted viewer provides one or more filters to one or more in-ear devices, and the one or more in-ear devices apply the one or more filters to the sound signal to generate an enhanced sound signal, which is then presented to the user.

[0040] Current audio enhancement techniques are ineffective at mitigating the cocktail party problem. For example, traditional hearing aids have a very small form factor, limiting their space for larger multi-microphone arrays. Furthermore, the small form factor imposes limitations on processing load and power consumption. Given this, it becomes clear why traditional hearing aids typically perform poorly in isolating and amplifying sound from sources, such as in crowded restaurants (i.e., they fail to truly alleviate the cocktail party problem). In contrast, the audio system described in this paper is part of a headset with a much larger form factor and can include a much larger microphone array, a larger power supply, and more powerful processing capabilities. Therefore, it can potentially allow for a very simple design for in-ear devices. The large microphone array, combined with the audio system's processing power, significantly enhances the ability to locate sound sources within localized areas (compared to traditional hearing aids), greatly alleviating the cocktail party problem and, in some cases, resolving it. Furthermore, the audio system can compensate for a user's hearing loss by generating audio content using the user's voice profile.

[0041] Embodiments of the present invention may include an artificial reality system or a combination thereof. An artificial reality is a form of reality that has been adjusted in some way before being presented to a user. This artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated by combining it with captured (e.g., real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single-channel or multi-channel manner (e.g., stereoscopic video providing a three-dimensional effect to the viewer). Furthermore, in some embodiments, the artificial reality may also be associated with applications, products, accessories, services, or some combination thereof for creating content in the artificial reality, and / or otherwise used in the artificial reality. Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted viewers) connected to a host computer system, standalone wearable devices (e.g., head-mounted viewers), mobile devices or computing systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0042] Figure 1AThis is a perspective view of a head-mounted viewer 100 implemented as an eyewear device according to one or more embodiments. In some embodiments, the eyewear device is a near-eye display (NED). Typically, the head-mounted viewer 100 can be worn on a user's face to present content (e.g., media content) using the display components and / or audio system in the head-mounted viewer 100. Examples of media content include one or more images, videos, audio, or some combination thereof. However, the head-mounted viewer 100 can also be used to present media content to a user in different ways. For example, audio content can be presented using one or both of the two in-ear devices 185 worn by the user. The head-mounted viewer 100 includes a frame 110 and may include a display component (which includes one or more display elements 120), one or more imaging devices 130, a depth camera assembly (DCA), a position sensor 190, a passive camera assembly (PCA), an eye-tracking system, and other components such as an audio system. Although Figure 1A The illustration shows example locations of components of the head-mounted viewer 100 on the head-mounted viewer 100, but these components may be located at other locations on the head-mounted viewer 100, on peripheral devices paired with the head-mounted viewer 100, or some combination thereof. Similarly, more components may be present on the head-mounted viewer 100 than... Figure 1A The number of components shown may be more or less.

[0043] Frame 110 holds the other components of the head-mounted viewer 100 described above. Frame 110 includes a front portion and end pieces (e.g., temples) that hold one or more display elements 120 and that are attached to the user's head. The front portion of frame 110 extends across the top of the user's nose. The length of the end pieces may be adjustable (e.g., adjustable temple length) to fit different users. End pieces may also include curled portions behind the user's ears (e.g., temple tips, eyeglass temples).

[0044] The one or more display elements 120 provide light to a user wearing the head-mounted viewer 100. As shown, the head-mounted viewer includes a display element 120 for each of the user's eyes. In some embodiments, the display element 120 generates image light that is provided to the eyebox of the head-mounted viewer 100. The eyebox is the location in the space occupied by the user's eyes when wearing the head-mounted viewer 100. For example, the display element 120 may be a waveguide display. A waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is internally coupled into the one or more waveguides, which output light in a manner that creates a pupil replication within the eyebox of the head-mounted viewer 100. The internal coupling of light and / or the external coupling of light from the one or more waveguides may be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a mirror, etc.) that scans the light as it is internally coupled into the one or more waveguides. It is important to note that in some embodiments, one or both of the two display elements 120 are opaque and do not transmit light from a local area surrounding the head-mounted viewer 100. This local area is the area surrounding the head-mounted viewer 100. For example, this local area could be the room where a user wearing the head-mounted viewer 100 is located, or the user wearing the head-mounted viewer 100 might be outside and the local area is an outdoor area. In this case, the head-mounted viewer 100 generates VR content. Alternatively, in some embodiments, one or both of the two display elements 120 are at least partially transparent, such that light from the local area can be combined with light from the one or more display elements to generate AR content and / or MR content.

[0045] In some embodiments, the display element 120 does not generate image light, but instead transmits light from a localized area to a lens within the eye-movement range. For example, one or both of the two display elements 120 may be an uncorrected lens (over-the-counter lens) or a prescription lens (e.g., a single-vision lens, bifocal and trifocal lens, or progressive lens) to help correct the user's visual impairment. In some embodiments, the display element 120 may be polarized and / or tinted to protect the user's eyes from sunlight.

[0046] It should be noted that in some embodiments, display element 120 may include additional optical modules (not shown). The optical modules may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that guide light from display element 120 into the eye-tracking range. The optical modules may, for example, correct aberrations in some or all portions of the image content, magnify some or all portions of the image, or some combination thereof.

[0047] In some embodiments, the head-mounted viewer 100 may include one or more imaging devices 130 that acquire visual information about a local area surrounding the head-mounted viewer 100. In some embodiments, the imaging devices 130 are used by a depth camera assembly (DCA). The DCA determines depth information about a portion of a local area surrounding the head-mounted viewer 100. The DCA includes the one or more imaging devices 130 and a DCA controller (…). Figure 1A (Not shown in the image), and may also include an illuminator 140. In some embodiments, the illuminator 140 uses light to illuminate a portion of a local area. This light may be, for example, infrared (IR) structured light (e.g., dotted patterns, multiple stripes, etc.), an infrared flash for time-of-flight, etc. In some embodiments, the one or more imaging devices 130 acquire an image of a portion of the local area including light from the illuminator 140. As shown, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, the illuminator 140 is absent and at least two imaging devices 130 are present.

[0048] The DCA controller uses the acquired images and one or more depth determination techniques to calculate the depth information of a local area. The depth determination techniques can be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using textures added to the scene by light from illuminator 140), some other technique for determining the depth of the scene, or some combination thereof.

[0049] Position sensor 190 generates one or more measurement signals and estimates position information about head-mounted viewer 100. Position information may include the position of head-mounted viewer 100 in a local area, the orientation of head-mounted viewer 100, the velocity of head-mounted viewer 100, or some combination thereof. The position information of head-mounted viewer 100 worn by a user corresponds to position information about the user's head. Position sensor 190 may be located on a portion of frame 110 of head-mounted viewer 100. Position sensor 190 may include an inertial measurement unit (IMU). Examples of position sensor 190 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting motion, a type of sensor for error correction of the IMU, or some combination thereof. Position sensor 190 may be located outside the IMU, inside the IMU, or some combination thereof.

[0050] In some embodiments, the head-mounted viewer 100 may provide simultaneous localization and mapping (SLAM) for the position of the head-mounted viewer 100 and for updating the model of a local region. For example, the head-mounted viewer 100 may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more imaging devices 130 (e.g., RGB cameras) that acquire images of some or all portions of the local region. The images acquired by the PCA and the depth information determined by the DCA may be used to determine positional information about one or more sound sources in the local region (i.e., the location of each sound source within the local region), to generate a model of the local region including the position of each sound source, to update the model of the local region over time (i.e., to update the model as one or more sound sources change position), or some combination thereof. Furthermore, a position sensor 190 tracks the position (e.g., position, orientation, and velocity) of the head-mounted viewer 100 within the model of the local region. In some embodiments, the model of the local region is stored in a head-mounted viewer (e.g., in an audio system), in an external system (e.g., in a map building server), in a mobile device, or in any combination thereof.

[0051] In some embodiments, PCA can use an object recognition model to identify the sound source type of each real-world sound source in a local area. The sound source type is the category of the entity emitting sound in the local area. For example, PCA can use an object recognition model to identify the type of real-world sound source. A real-world sound source can be, for example, a person, a person wearing a head-mounted viewer, a speaker, an animal, a machine, some other real-world entity emitting sound in the local area, or some combination thereof. PCA can update the model of the local area to include the type of each sound source. PCA can also update the model of the local area by tracking gestures made by each person or person wearing a head-mounted viewer. Gestures can include speaking, waving, raising a hand, handing a real-world object to a user, or some other gesture made by a person or person wearing a head-mounted viewer.

[0052] In some embodiments, PCA can acquire images of the user. Images of the user acquired by PCA can be used, along with gestures made by the user, to update a model of a local region. A gesture is any movement made by the user that represents a command (i.e., implicit user input). User gestures can include, for example, indicating gestures of one or both hands, indicating gestures of one or more fingers, indicating gestures of one or both arms, some other movement that represents a command, or some combination thereof.

[0053] In some embodiments, the head-mounted viewer 100 may include one or more imaging devices 130 that acquire one or more images of one or both eyes of a user wearing the head-mounted viewer 100. In some embodiments, the imaging devices 130 are used by an eye-tracking system to determine eye-tracking information associated with the user's eyes based on the acquired images. The eye-tracking system may also include one or more light sources (not shown) that illuminate the eyes using IR light. The light source may be, for example, a vertical-cavity surface-emitting laser, a light-emitting diode (LED), a micro-LED, some other infrared (IR) light source, or some combination thereof. The one or more imaging devices 130 used by the eye-tracking system are configured to acquire images of one or both eyes illuminated using IR light from one or more light sources.

[0054] Eye-tracking systems can use acquired images and depth determination techniques to determine eye-tracking information for one or both eyes of a user. Depth determination techniques may include, for example, structured light, time-of-flight, stereo imaging, or other depth determination methods familiar to those skilled in the art. In some embodiments, eye-tracking sensors determine eye-tracking information based on acquired images and a model of the user's eyes. The eye-tracking information determined by the eye-tracking system may include, for example, gaze information, pupillary measurement information, or some combination thereof. The user's gaze is the direction the user is looking in (e.g., the user's eye orientation). The user's pupillary measurement information includes pupil size and pupillary reactivity. One or both of the gaze information and pupillary measurement information can be used to determine where the user is directing their attention (e.g., implicit user input).

[0055] Eye-tracking information determined by the eye-tracking system can be combined with the determined position of the head-mounted viewer 100 (e.g., as determined using position sensor 190) to determine the user's gaze. For example, a vector corresponding to the user's eye orientation (indicating the orientation of the user's eyes relative to the user's head) can be added to a vector corresponding to the position of the head-mounted viewer (indicating the position and orientation of the head-mounted viewer within a local area) to determine a vector corresponding to the user's gaze (indicating the direction the user is looking in the local area). The model of the local area can include the user's gaze vector. In some embodiments, the model of the local area can include a history of mapped gaze vectors of the user. By determining the user's gaze and identifying the direction the user is looking in the local area, this can be combined with knowledge of the positions of other sound sources in the local area (e.g., as determined by the audio system) to allow determination of which sound source the user is looking at.

[0056] In some embodiments, the eye-tracking system may receive one or more images from the PCA and map a determined gaze to a location within the received images, wherein the one or more images depict a local region within the field of view of one or more imaging devices 130 used by the PCA. The model of the local region may be updated to include the gaze.

[0057] The audio system provides audio content to a user via a head-mounted viewer 100 and / or via an in-ear device 185. The audio system includes a microphone array, an audio controller 150, and a transducer array. However, in other embodiments, the audio system may include different components and / or additional components. Similarly, in some cases, the functions described for multiple components in the reference audio system may be distributed among these components in a manner different from that described herein. For example, some or all of the multiple functions of the controller may be performed by a remote server.

[0058] A microphone array detects sound within a localized area of ​​the head-mounted viewer 100. Microphones 180 collect sound emitted from one or more real-world sound sources within the localized area (e.g., a room). The microphone array includes multiple microphones 180. Each microphone 180 can be a sound wave sensor, a sound transducer, or a similar sensor suitable for sound detection. The number and / or location of the microphones 180 can be... Figure 1AThe number and / or positions of the microphones shown vary. For example, the number of microphone positions can be increased to increase the amount of audio information collected and improve the sensitivity and / or accuracy of that information. The microphone positions can be oriented such that microphone 180 can detect sound in a wide range of directions around a user wearing head-mounted viewer 100. Each microphone 180 is configured to detect sound and convert the detected sound into an electronic format (analog or digital). In some embodiments, microphone 180 may be placed on the outer surface of head-mounted viewer 100, on the inner surface of head-mounted viewer 100, separate from head-mounted viewer 100 (e.g., as part of some other device), or some combination thereof.

[0059] In one embodiment, the microphone array collects leakage signals emitted by the in-ear device 185. In other embodiments, the microphone array may collect leakage signals emitted from the transducer array of the audio system of the head-mounted viewer 100. The leakage signal is an audio signal in phase with an enhanced sound signal presented to the user as audio content via the in-ear device 185 or via the audio system of the head-mounted viewer 100. The leakage signal can be heard by others or collected by other head-mounted viewer devices in a localized area.

[0060] The transducer array of the head-mounted viewer 100 can provide one or more sound signals to a user according to instructions from the audio controller 150. For example, the transducer array of the head-mounted viewer 100 can present an enhanced sound signal to the user according to instructions from the audio controller. The transducer array includes multiple transducers. The transducers can be speakers 160 or tissue transducers 170 (e.g., bone conduction transducers or cartilage conduction transducers). The number and / or position of the speakers 160 can be... Figure 1A The number and / or location of the speakers shown may vary. For example, speaker 160 may be mounted within the frame 110 of the head-mounted viewer 100. In some embodiments, instead of separate speakers for each ear, the head-mounted viewer 100 includes a speaker array comprising multiple speakers integrated into the frame 110 to improve the directionality of the presented audio content. Tissue transducers 170 are attached to the user's head and directly vibrate the user's tissues (e.g., bone or cartilage) to generate sound. The number and / or location of the transducers may vary. Figure 1A The number and / or location of the transducers shown may differ.

[0061] The transducer array can output a mitigation signal according to instructions from the audio controller 150. The mitigation signal essentially cancels out the leakage signal emitted by the in-ear device 185 in the far field, so that other people or head-mounted displays in the local area cannot hear or detect the leakage signal in the far field.

[0062] Audio controller 150 controls the operation of the audio system. Audio controller 150 may include a processor and a computer-readable storage medium. Audio controller 150 may identify a sound source (real-world or virtual) in a local area as a target sound source. In some embodiments, audio controller 150 identifies the target sound source based on implicit user input stored in a model of the local area. In some embodiments, audio controller 150 identifies the target sound source based on explicit user input (e.g., user-provided verbal cues).

[0063] Audio controller 150 can generate one or more filters based on a user's voice profile to apply to one or more sound signals acquired by a microphone array. These filters can be applied to, for example, sound signals corresponding to sounds emitted by a target sound source, sound signals corresponding to sounds emitted by other sound sources, or some combination thereof. In some embodiments, audio controller 150 can use a model of a local region to generate one or more filters. In some embodiments, these filters can enhance (e.g., by amplifying certain frequencies, by equalizing audio, etc.) the sound signal corresponding to the sound emitted by the target sound source. In some embodiments, audio controller 150 generates an enhanced signal by applying the one or more filters to the sound signal corresponding to the sound emitted by the target sound source. In some embodiments, audio controller 150 provides instructions to in-ear device 185 to apply one or more filters to the sound signal to generate an enhanced signal.

[0064] Audio controller 150 can generate one or more filters that suppress (e.g., by filtering out certain frequencies) one or more sound signals corresponding to sounds emitted by one or more other sound sources in a local area. In one embodiment, audio controller 150 determines a suppression signal. The suppression signal is a sound signal out of phase with a leakage signal emitted by in-ear device 185. The leakage signal may be heard by others and / or picked up by other devices (e.g., head-mounted displays) in the local area, thus compromising user privacy. In some embodiments, the leakage signal may be picked up by a microphone array. The audio controller generates a filter that filters out frequencies of the leakage signal. The audio controller applies this filter to the leakage signal to generate the suppression signal. The suppression signal is provided as an output signal by audio controller 150 to the transducer array. The suppression signal suppresses the leakage signal in the far field, thus providing greater privacy for the user.

[0065] In some embodiments, the audio controller 150 may generate one or more filters that suppress sound signals corresponding to sounds emitted by sound sources determined not to be the target sound source. Suppression of sound signals corresponding to sounds emitted by other sound sources helps resolve the cocktail party problem. For example, by suppressing frequencies of sound signals associated with other sound sources (e.g., multiple people in a crowd), the audio controller 150 can amplify the sound emitted by the target sound source (e.g., the person the user is talking to), even without applying any filters to the sound signals associated with the target sound source. In another example, the audio controller 150 can resolve the cocktail party problem by suppressing frequencies in sound signals associated with other sound sources and amplifying frequencies in sound signals associated with the target sound source. In some embodiments, the audio controller 150 may determine that frequencies in one or more sound signals associated with one or more other sound sources may be similar to frequencies in sound signals associated with the target sound source. In this embodiment, the audio controller 150 may determine not to amplify sound signals associated with other sound sources, but only to amplify the sound signal associated with the target sound source.

[0066] In some embodiments, the in-ear device 185 provides an enhanced sound signal to a user according to instructions from the audio controller 150. The in-ear device 185 may include a communication unit (not shown) and a speaker 165. The in-ear device 185 is worn by a user (e.g., placed in the user's corresponding ear canal) and can present the enhanced sound signal to the user via the speaker 165. In some embodiments, the in-ear device 185 is a hearing aid communicatively coupled to a head-mounted viewer 100. In some embodiments, the in-ear device 185 applies one or more filters to a sound signal to generate an enhanced sound signal according to instructions from the audio controller 150. The in-ear device 185 can then present the enhanced sound signal to the user via the speaker 165. Although Figure 1A One in-ear device 185 is shown for each of the user's ears, but in other embodiments, there may be only a single in-ear device 185 for one ear, and no in-ear device for the other ear.

[0067] about Figure 2 Additional details regarding the audio system are discussed below, and additional details regarding the components of the head-mounted viewer 100 are combined below. Figure 5 To elaborate.

[0068] Figure 1B This is a perspective view of a head-mounted viewer 105 implemented as an HMD according to one or more embodiments. In embodiments describing AR and / or MR systems, multiple portions of the front of the HMD are at least partially transparent to the visible wavelength range (~380 nm to 750 nm), and multiple portions of the HMD located between the front of the HMD and the user's eyes are at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a band 175. The head-mounted viewer 105 includes the components referenced above. Figure 1A The described components are some of the same components, but these components are modified to be integrated with the HMD shape factor. For example, the HMD includes one or more imaging devices 130, a depth camera assembly (DCA), a position sensor 190, a passive camera assembly (PCA), an eye-tracking system, and an audio system. Figure 1B An illuminator 140, multiple speakers 160, multiple microphones 180, and an in-ear device 185 with a speaker 165 are shown. Although Figure 1B One in-ear device 185 is shown, but in other embodiments, two in-ear devices 185 may be present, with one in-ear device used for each ear.

[0069] Figure 2 This is a block diagram of an audio system 200 according to one or more embodiments. Figure 1A or Figure 1B The audio system in the audio system can be an embodiment of audio system 200. Figure 2 In some embodiments, the audio system 200 includes a microphone array 210, a transducer array 220, and an audio controller 230. In some embodiments, the audio system 200 also includes an in-ear device 270. Some embodiments of the audio system 200 have components different from those described herein. Similarly, in some cases, multiple functions may be distributed among the components in a manner different from that described herein. At least a portion of the audio system 200 is integrated into a head-mounted viewer, and in some cases, the audio system 200 may also include an in-ear device 270. Additionally, in conjunction with... Figure 2 The functions described by one or more of the components shown may be distributed across one or more external components. For example, some or all of the functions of the audio controller 230 may be performed by a connected mobile device (e.g., a mobile phone).

[0070] Microphone array 210 collects sound from a localized area surrounding it. Microphone array 210 may include multiple acoustic sensors, each detecting changes in air pressure of sound waves and converting the detected sound into an electronic format (analog or digital). These multiple acoustic sensors may be located on a head-mounted display (e.g., head-mounted display 100 and / or head-mounted display 105), on an in-ear device 270, on some other connected device (e.g., a mobile phone), or on some combination thereof. The acoustic sensors may be, for example, microphones, vibration sensors, accelerometers, or any combination thereof. Increasing the number of acoustic sensors can improve the accuracy of information describing the sound field (e.g., directionality) generated by any sound source.

[0071] In one embodiment, the microphone array may acquire sound emitted by one or more real-world sound sources within a local area. The acquired sound may include multiple sound signals (i.e., sound signals from each sound source). These sound signals are provided to the audio controller 230 for further processing, such as determining one or more filters that may amplify at least one sound signal before presenting it to the user as audio content. In some embodiments, the acquired sound may include leakage signals emitted by a transducer (e.g., internal transducer 275) of the in-ear device 270 or by one or more transducers in the transducer array 220.

[0072] In some embodiments, transducer array 220 provides one or more sound signals to a user. Transducer array 220 includes a plurality of transducers. Transducers may be, for example, loudspeakers (e.g., loudspeaker 160), tissue transducers (e.g., tissue transducer 170), some other device for presenting sound signals to a user, or some combination thereof. Tissue transducers may be configured to function as bone conduction transducers or cartilage conduction transducers. Transducer array 220 may present audio content via air conduction (e.g., via one or more loudspeakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, transducer array 220 may include one or more transducers to cover different portions of a frequency range. For example, piezoelectric transducers may be used to cover a first portion of the frequency range, and moving coil transducers may be used to cover a second portion of the frequency range.

[0073] A bone conduction transducer generates sound pressure waves by vibrating the bone / tissue in the user's head. The bone conduction transducer can be coupled to a portion of a head-mounted display and can be configured to be located behind the auricle (which is attached to a portion of the user's skull). The bone conduction transducer receives vibration commands from an audio controller 230 and, based on the received commands, causes a portion of the user's skull to vibrate. The vibrations from the bone conduction transducer generate sound pressure waves that are applied to the tissue and propagate around the eardrum toward the user's cochlea.

[0074] A cartilage transducer generates sound pressure waves by vibrating one or more portions of the auricular cartilage of the user's ear. The cartilage transducer can be coupled to a portion of a head-mounted display and can be configured to attach to one or more portions of the auricular cartilage of the ear. For example, the cartilage transducer can be attached to the back of the auricle of the user's ear. The cartilage transducer can be located anywhere along the auricular cartilage around the outer ear (e.g., the auricle, tragus, other portion of the auricular cartilage, or a combination thereof). Vibrating one or more portions of the auricular cartilage can generate: sound pressure waves propagating through the air outside the ear canal; sound pressure waves generated by tissue that causes some portions of the ear canal to vibrate, thereby generating air-propagating sound pressure waves within the ear canal; or a combination thereof. The generated air-propagating sound pressure waves propagate along the ear canal toward the eardrum.

[0075] In some embodiments, transducer array 220 may present an enhanced sound signal as audio content to a user based on instructions provided by audio controller 230. In these embodiments where audio content is provided solely through transducer array 220 of a head-mounted viewer, the transducer array may include one or more dipole speaker assemblies to suppress leakage signals. The construction and operation of dipole speaker assemblies are further described, for example, in U.S. Patent Application No. 16 / 737,316, filed January 8, 2020; U.S. Patent Application No. 16 / 555,058, filed August 29, 2019; and U.S. Patent Application No. 16 / 790,613, filed February 13, 2020, the contents of which are incorporated herein by reference in their entirety. In other embodiments, transducer array 220 may receive instructions from audio controller 230 for outputting a suppression signal. In these embodiments, audio content is provided solely through in-ear device 270.

[0076] In some embodiments, the in-ear device 270 may present an enhanced sound signal as audio content to the user according to instructions provided by the audio controller 230. The in-ear device 270 may be... Figure 1A or Figure 1B An embodiment of the in-ear device 185 is described. Each in-ear device 270 includes at least a communication unit 273 and an internal transducer 275. In some embodiments, one or more in-ear devices 270 may include a battery, an external transducer, an in-ear audio controller, an external microphone, an internal microphone, some other components not discussed herein, or some combination thereof. In some embodiments, one or more of these in-ear devices 270 may be, for example, earbuds communicatively coupled to the audio system 200. The communication unit 273 includes a receiver, and in some embodiments, the communication unit may also include a transmitter. In some embodiments, the communication unit 273 may provide sound acquired by the external microphone and / or the internal microphone to the audio controller 230. The communication unit 273 may receive instructions from the audio controller 230. These instructions may include one or more filters to be applied to a sound signal emitted from a target sound source and / or to enhance the sound signal. In some embodiments, the in-ear audio controller may apply one or more filters to the sound signal to generate an enhanced sound signal. The internal transducer 275 presents the enhanced sound signal to the user. Each in-ear device 270 is configured to be properly inserted into the user's ear canal such that it at least partially closes the ear canal.

[0077] During the presentation of the enhanced sound signal to the user, the in-ear device 270 may emit a leakage signal. This leakage signal may be heard by others and / or picked up by other devices (e.g., head-mounted viewers) in a localized area, thus compromising the user's privacy. In some embodiments, the leakage signal may be monitored by the microphone array 210, and the suppression signal output by the transducer array 220 in the far field is substantially suppressed. This is in Figure 1A This was discussed in further detail in the text.

[0078] The audio controller 230 controls the operation of the audio system 200. Figure 2 In some embodiments, the audio controller 230 includes a data storage 235, a calibration module 237, a direction of arrival (DOA) estimation module 240, a transfer function module 245, a tracking module 250, a source identification module 255, a sound filter module 260, and a communication module 265. In some embodiments, the audio controller 230 may be located inside the head-mounted viewer. Some embodiments of the audio controller 230 have components different from those described herein. Similarly, various functions may be distributed among the components in a manner different from that described herein. For example, some functions of the audio controller 230 may be performed outside the head-mounted viewer.

[0079] Data storage 235 stores data used by audio system 200. The data in data storage 235 may include recorded sound in a local region of audio system 200, direction of arrival (DOA) estimates, sound source locations, target sound sources, head-related transfer functions (HRTF), transfer functions of one or more sensors, array transfer functions (ATF) of one or more acoustic sensors, a model of the local region, user input, one or more audiograms of the user, user speech test results under noise, user spectral-time discrimination results, user voice profiles, multiple sound filters, multiple sound signals, other data related to use by audio system 200, or any combination thereof.

[0080] The model of a local region tracks the position, motion, and sound of each sound source (including the user) within that local region and stores acoustic parameters describing the local region. The model may include location information about the user (e.g., the user's position, orientation, and / or velocity within the local region), motion information about the user (e.g., eye-tracking information and / or gestures made by the user), and acoustic information about the user (e.g., one or more filters used to amplify or suppress sounds emitted by the user). The model may also include location information about sound sources (e.g., the position of each sound source within the local region), type information about some or all of the sound sources (e.g., sound sources are people, sound sources are virtual objects, etc.), and acoustic parameters describing the local region (e.g., reverberation time). In some embodiments, the audio system updates the model of the local region over time using updated information about the user and / or updated information about the sound sources and / or updated information about the local region.

[0081] The calibration module 237 generates a user's voice profile. The voice profile is personalized voice information about the user, describing their hearing response to sounds at different frequencies. The voice profile may include information from one or more audiograms, loudness discomfort level test results, speech test results under noise, spectral-time discrimination results, temporal fine structure test results, cognitive performance test results, self-reported hearing ability, some other hearing test results, or some combination thereof. In some embodiments, the voice profile may include information from one or more electroencephalogram (EEG) tests, pupillary measurements, cardiac activity parameters, or some other biosignal test results of the user. In some embodiments, the calibration module 237 generates the voice profile by presenting audio content to the user via transducer array 220 and / or in-ear device 270. The audio content is presented at different frequencies (e.g., using pure tones), and the user provides feedback on whether / when they heard the audio content. In some embodiments, a more complex spectral time-varying signal is used to present the audio content. In some embodiments, a user may provide feedback via an I / O interface communicatively coupled to the audio system 200. In other embodiments, a user may make a gesture indicating whether / when they have heard the audio content (e.g., giving a thumbs-up), which is captured by the PCA of a head-mounted viewer. In some embodiments, a user may provide verbal cues as feedback, which are captured by the microphone array 210. In some embodiments, a calibration module 237 collects sound profiles from the user and / or from an audiologist via an I / O interface or some other external system communicatively coupled to the audio system 200.

[0082] The calibration module 237 can update the user's voice profile as needed over time. For example, during operation of the audio system 200, the calibration module can receive feedback from the user related to the performance of the head-mounted viewer and / or in-ear device 270. User feedback may include feedback related to the presented audio content (e.g., posture or speech cues). Based on the received feedback, the calibration module 237 can update the user's voice profile accordingly. In some embodiments, user feedback may be less obvious. For example, a user may tend to turn up the volume when the presented audio content is in a specific frequency band (e.g., at higher frequencies). Over time, the calibration module 237 can identify this pattern and adjust the voice profile to reflect hearing loss in that specific frequency band. In another example, a user may move their head and / or eyes more frequently when the presented audio content is in a specific frequency band. Over time, the calibration module 237 can identify this pattern and adjust the voice profile to reflect hearing loss in that specific frequency band. The updated voice profile can then be used to generate a filter that takes into account hearing loss in the specific frequency band.

[0083] DOA estimation module 240 is configured to locate sound sources in a local area, in part based on sound acquired from microphone array 210. Localization is the process of determining the position of a sound source relative to the user of audio system 200. DOA estimation module 240 performs DOA analysis to locate one or more sound sources in the local area and updates the model of the local area accordingly. DOA analysis may include analyzing the intensity, spectrum, and / or time of arrival of each sound at microphone array 210 to determine the direction of sound source. In some cases, DOA analysis may include any suitable algorithm for analyzing the surrounding acoustic environment in which audio system 200 is located.

[0084] For example, DOA analysis can be designed to receive input signals from microphone array 210 and apply digital signal processing algorithms to these input signals to estimate the direction of arrival (DOA). These algorithms may include, for example, a delay-sum algorithm, in which the input signal is sampled and multiple variations of the sampled signal, obtained by weighting and delaying them, are averaged to determine the DOA. A least mean squared (LMS) algorithm can also be implemented to generate an adaptive filter. This adaptive filter can then be used to identify, for example, differences in signal strength or differences in arrival time. These differences can then be used to estimate the DOA. In another embodiment, the DOA can be determined by converting the input signal to the frequency domain and processing multiple specific frequency intervals (bins) within the time-frequency (TF) domain. Each selected TF frequency interval can be processed to determine whether the frequency interval includes a portion of the audio spectrum containing a direct-path audio signal. Those frequency intervals containing a portion of the direct-path audio signal can then be analyzed to determine the angle at which microphone array 210 receives the direct-path audio signal. The determined angle can then be used to identify the DOA of the received input signal. Other algorithms not listed above can also be used alone or in combination with the algorithms above to determine the DOA.

[0085] The transfer function module 245 is configured to generate one or more acoustic transfer functions. Typically, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function module 245 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. An ATF characterizes how a microphone (e.g., a microphone in microphone array 210 and / or an external microphone in in-ear device 270) receives sound from a point in space.

[0086] The ATF comprises multiple transfer functions that characterize the relationship between multiple sound sources and the corresponding sounds received by the multiple acoustic sensors in the microphone array 210. Therefore, for a single sound source, there exists a corresponding transfer function for each acoustic sensor in the microphone array 210. This set of transfer functions is collectively referred to as the ATF. Thus, for each sound source, there exists a corresponding ATF. Because human physiology (e.g., ear shape, shoulders, etc.) affects the sound as it travels towards the ear, the ATF relative to a specific sound source location in the microphone array 210 may vary from user to user. Therefore, these ATFs of the microphone array 210 are personalized for each user of the audio system 200.

[0087] In some embodiments, the transfer function module 245 determines one or more HRTFs for a user of the audio system 200. HRTFs characterize how an ear receives sound from a point in space. Because a person's physiological structure (e.g., ear shape, shoulders, etc.) affects how sound propagates toward both ears, the HRTF relative to a person's specific sound source location is unique to each ear (and thus unique to that person). In some embodiments, the transfer function module 245 may use a calibration process to determine the user's HRTF. In some embodiments, the transfer function module 245 may provide information about the user to a remote system. This remote system uses, for example, machine learning to determine a set of HRTFs tailored for the user and provides this tailored set of HRTFs to the audio system 200.

[0088] Tracking module 250 is configured to track the location of one or more sound sources. Tracking module 250 can compare multiple current DOA estimates and compare them to a stored history of previous DOA estimates. In some embodiments, audio system 200 can recalculate DOA estimates according to a periodic schedule, e.g., once per second or once per millisecond. Tracking module 250 can compare current DOA estimates with previous DOA estimates, and in response to changes in the DOA estimates of sound sources, tracking module 250 can determine that a sound source has moved. In some embodiments, tracking module 250 can detect changes in location based on visual information received from a head-mounted viewer (e.g., from DCA and / or PCA) or some other external source. Tracking module 250 can track the movement of one or more sound sources over time. Tracking module 250 can store the number of sound sources and the location of each sound source in a model of a local region at each time point. In response to changes in the number or location of sound sources, tracking module 250 can determine that a sound source has moved, and the model of the local region is updated accordingly. Tracking module 250 can calculate an estimate of localization variance. Local variations can be used as the confidence level for each determination of motion variations.

[0089] Source identification module 255 is configured to determine a sound source (e.g., a target sound source) of interest to the user at any given time from among a plurality of sound sources in a local region. The target sound source can be a real-world sound source or a virtual sound source. In one embodiment, source identification module 255 determines the target sound source based in part on implicit user input stored in a model of the local region and / or explicit user input directly provided by the user. For example, the model of the local region includes location information about the user, location information about the sound source, and eye-tracking information about the user. Based on the mapped gaze vector (i.e., implicit user input), source identification module 255 determines that the target sound source is the sound source the user is looking at. In some embodiments, the user's gaze is tracked over time in the model of the local region, and the target sound source is the sound source the user is looking at and that lasts for a threshold amount of time (e.g., for 3 seconds, for 5 seconds, etc.). In another example, the model of the local region includes gesture information about the user (i.e., implicit user input). Based on the identified gesture made by the user (e.g., pointing with a finger), source identification module 255 determines that the target sound source is the sound source the user is pointing at. In some embodiments, the source identification module 255 determines the target sound source based on explicit user input. For example, the user can provide a verbal prompt (e.g., stating which sound source should be the target sound source), which is captured by a microphone array. Based on the user input, the source identification module 255 determines that the target sound source is the sound source indicated by the user.

[0090] Sound filter module 260 determines one or more filters to apply to one or more sound signals. These sound signals may correspond to sounds emitted by a target sound source and / or other sound sources (e.g., leakage sound emitted by in-ear device 270). In some embodiments, the one or more sound filters may enhance the sound signal associated with the target sound source. In some embodiments, sound filter module 260 determines one or more sound filters that filter out (i.e., reduce or eliminate) sounds associated with other sound sources in a local area. In some embodiments, sound filter module 260 may determine that it may be necessary to spatialize the sound signal emitted by the target sound source using filters (e.g., one or more HRTFs determined by transfer function module 245). In some embodiments, sound filter module 260 may determine one or more filters (e.g., multi-band limiters) to prevent discomfort associated with high sound pressure levels at specific frequencies. In some embodiments, sound filter module 260 may identify one or more filters for suppressing feedback. For example, a frequency conversion filter that transposes all or some frequency bands by a specific amount (e.g., transposes 5 Hz, 20 Hz, 50 Hz, etc.) may be identified.

[0091] The sound filter module 260 can determine one or more filters based on a user's voice profile. For example, the sound filter module 260 can select a filter to amplify certain frequencies based on the user's voice profile, which includes information about certain frequencies that are difficult for the user to hear. In another example, the sound filter module 260 can select a wide dynamic range compression filter from a plurality of wide dynamic range compression filters based on the user's voice profile, which includes information about how strong the sound needs to be for the user to hear it. The wide dynamic range compression filter applies a non-linear gain to the sound signal. For example, the wide dynamic range compression filter can amplify soft sounds (i.e., low-intensity sounds) significantly, moderate sounds (i.e., medium-intensity sounds) moderately, and loud sounds (i.e., high-intensity sounds) slightly. In another example, the wide dynamic range compression filter can amplify the sound signal based on frequency to equalize the sound presented to the user. Each unique sound signal can have its own unique wide dynamic range compression filter. The sound filter module 260 determines which wide dynamic range compression filter is suitable for application to the sound signal associated with the target sound source.

[0092] In some embodiments, the sound filter module 260 generates a filter based on a model of a local region. The model of the local region may include multiple types of sound sources within the local region. For example, at least one sound source in the local region (i.e., not a target sound source) may be a mechanical device. The sound filter module 260 may determine a sound filter for reducing or eliminating the frequency of sound emitted by the mechanical device. The model of the local region may include multiple acoustic parameters describing multiple acoustic characteristics of the local region. These acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the sound filter module 260 calculates one or more of these acoustic parameters and stores them in the model of the local region. In some embodiments, the sound filter module 260 requests a map from a map building server (e.g., as described below regarding...). Figure 5 The sound filter module 260 can determine one or more sound filters based on the high reverberation time of the local region, which reduce or eliminate the frequencies of sounds emitted by other sound sources (i.e., sounds not emitted by the target sound source).

[0093] The sound filter module 260 can detect that the identified target sound source is located within a threshold distance of a different sound source in a local region. For example, the sound filter module 260 can use determined locations of one or more sound sources and a model of the local region to determine whether the identified target sound source is located within a threshold distance of a different sound source. The threshold distance can be unique for each sound source. For example, the threshold distance of a sound source can be based on the frequency of the sound emitted by that sound source, the intensity of the sound emitted by that sound source, and the user's voice profile, or some combination thereof. For example, in order for the user to hear certain frequencies and / or certain intensities, these frequencies and / or these intensities may need to be further separated. The sound filter module 260 updates at least one of one or more filters (e.g., by updating the HRTF) to increase the spatial separation between the target sound source and the different sound source to at least an interval distance greater than the threshold distance. In this embodiment, the enhanced sound signal makes the sound sound as if it originates from the target sound source, and the sound from the different sound source sounds as if it originates from a location in the local region at least at the interval distance from the target sound source. Figure 3 This embodiment is described in more detail below.

[0094] The sound filter module 260 can apply one or more filters to the sound signal to generate an enhanced sound signal. In some embodiments, the enhanced sound signal can be provided to the transducer array 320 for presentation to a user. In other embodiments, the enhanced sound signal can be provided to the communication module 265. In some embodiments, the in-ear device 270 applies one or more filters to the sound signal.

[0095] Communication module 265 communicates with one or more external systems communicatively coupled to audio system 200. Communication module 265 may include a receiver (e.g., an antenna) and a transmitter. External systems may include, for example, other components in a head-mounted viewer, an artificial reality console, an I / O interface, a map building server, in-ear device 270, etc. Communication module 265 can send and receive data related to a model of a local area with respect to the map building server. Communication module 265 can send data (e.g., one or more filters, enhanced sound signals) and receive data (e.g., captured sound) with respect to the communication unit 273 in in-ear device 270. Communication module 265 can receive data (e.g., user voice profile, user feedback) from external systems (e.g., I / O interfaces).

[0096] The processing and calculations performed by the audio controller 230 allow for better enhancement of the audio content presented to the user. For example, a user with hearing loss benefits from the audio controller 230, which identifies a target sound source (emitting sounds at frequencies that the user cannot hear) and generates one or more filters to apply to the sound emitted by that target sound source, amplifying those specific frequencies. Sound sources located in localized areas are better tracked by the microphone array 210 of the head-mounted viewer, allowing for better identification of target sound sources. Moreover, the processing and calculations performed by the audio controller 230 allow the in-ear device 270 to present more personalized and enhanced audio content to the user.

[0097] Figure 3 This is an example of a local region 300 including one or more sound sources according to one or more embodiments. The local region 300 includes a user 310 wearing a head-mounted viewer 320, located near multiple sound sources (e.g., real-world sound source 340, real-world sound source 350, real-world sound source 360, and virtual sound source 390). Real-world sound source 340 is depicted as a mechanical device, real-world sound source 350 is depicted as a person, real-world sound source 360 ​​is depicted as another person, and virtual sound source 390 is depicted as an alien. In some embodiments, the head-mounted viewer 320 may be... Figure 1A Head-mounted viewer 100 or Figure 1BThe head-mounted viewer 105 is shown in the image. In one embodiment, the user 310 may also wear an in-ear device (not shown), which may be... Figure 1A and Figure 1B An embodiment of the in-ear device 185 is described. The multiple sound sources can collectively produce sound, which may be unclear when a user 310 with some degree of hearing loss hears all the sounds at once. In some embodiments, the user 310 (with or without hearing loss) may not be able to distinguish the sound produced by real-world sound source 350 from the sound produced by real-world sound source 360. The audio system can generate one or more filters to apply to the sounds emitted by any or all of the sound sources 340, 350, 360, and 390, thereby enhancing the audio content presented to the user 310.

[0098] The audio system of the head-mounted viewer 320 (e.g., audio system 200) is based on a model of local region 300 or based on the above reference. Figure 2 The user input described is used to identify the target sound source (e.g., sound source 350). Figure 3 In the illustrated embodiment, the audio system may determine the target sound source based on implicit feedback provided by the user, included in the model of local region 300 (e.g., based on gaze vector 330). The gaze vector 330 is determined to point to a location of real-world sound source 350, and the audio system identifies real-world sound source 350 as the target sound source. The audio system of the head-mounted viewer 320 determines one or more filters that can be applied to audio content presented by the head-mounted viewer 320 worn by the user 310 and / or an in-ear device (not shown). The audio system determines the filters based on the user 310's voice profile, the model of local region 300, or both.

[0099] The audio system can update at least one of one or more filters to provide better spatialization of the sound emitted by a target sound source. In the illustrated embodiment, a threshold distance 370 is depicted. The threshold distance 370 is a predetermined spherical distance measured outward from the center of the real-world sound source 350 (i.e., the target sound source in this example). The threshold distance 370 may depend on, for example, the frequency of the sound emitted by the real-world sound source 350, the amplitude of the sound emitted by the real-world sound source 350, the voice profile of the user 310, or some combination thereof. The audio system can detect that the real-world sound source 360 ​​is within the threshold distance 370 of the target sound source (i.e., the real-world sound source 350). The audio system updates at least one of one or more filters (e.g., by updating the HRTF) to increase the spatial spacing between the target sound source and the real-world sound source 360 ​​to at least an interval distance 380, which is greater than the threshold distance 370. In some embodiments, the updated filter may be applied by the audio system to the sound signal associated with the target sound source. In other embodiments, the updated filter can be applied by the in-ear device to the sound signal associated with the target sound source. The enhanced sound signal (i.e., the sound signal after the filter is applied) makes the sound appear to originate from the target sound source, and the sound from real-world sound source 360 ​​appears to originate from a location in a local region corresponding to the adjusted sound source 365. User 310 may now be able to distinguish between the two real-world sound sources 350 and 360, and in some embodiments, the user may hear real-world sound source 350 more easily.

[0100] Figure 4 A flowchart illustrating a process 400 for enhancing an audio signal to present it to a user, according to one or more embodiments, is provided. Figure 4 The process illustrated can be performed by multiple components of an audio system (e.g., audio system 200). In other embodiments, other entities can perform the process. Figure 4 Some or all of the steps in the process. Various embodiments may include different and / or additional steps, or perform these steps in a different order.

[0101] The audio system acquires sound from a local area. In one embodiment, the audio system uses one or more microphones coupled to a frame in a head-mounted viewer worn by a user to acquire sound.

[0102] The audio system identifies one or more sound sources within a 420° local area based on the acquired sound. The acquired sound (collected by a microphone) and acquired images (collected by the DCA and / or PCA of a head-mounted viewer) can be combined to determine a model of the local area. The model of the local area depicts the location of the one or more sound sources.

[0103] The audio system identifies a target sound source among 430 of the one or more sound sources. In some embodiments, the audio system determines the target sound source based on implicit user input stored in a model of a local region and / or explicit user input directly provided by the user. For example, the model of the local region may include a mapped gaze vector (i.e., implicit user input) used by the audio system when determining the target sound source. The mapped gaze vector may have been determined by the eye-tracking system of the head-mounted viewer. In some embodiments, the audio system may use explicit user input (e.g., a verbal cue from the user specifying the target sound source). This verbal cue may be acquired via a microphone.

[0104] An audio system determines one or more filters to apply to an audio signal associated with a target sound source. In some embodiments, the audio system determines the one or more filters based on a user's voice profile. The voice profile is personalized voice information about the user, describing how well the user hears different frequencies of sound. The voice profile may be stored in the audio system. Based on the voice profile, the audio system may determine to apply a filter that enhances the audio signal associated with the target sound source. In some embodiments, the audio system may determine to apply a filter based on a model of a local region, which better spatializes the audio signal associated with the target sound source. In some embodiments, the audio system may determine one or more filters to apply to other sounds associated with other sound sources located in the local region.

[0105] An audio system generates an enhanced audio signal by applying one or more filters to an audio signal. In one embodiment, the enhanced audio signal is made so that a sound that appears to originate from a target sound source can be distinguished from sounds emitted by other sound sources in a local area. For example, the enhanced audio signal makes a sound appear to originate from a target sound source, and a sound from a different sound source appears to originate from a location in the local area that is at least a distance away from the target sound source.

[0106] The audio system provides a 460° enhanced sound signal to a speaker assembly for presentation to the user. In one embodiment, the enhanced sound signal is provided to a speaker assembly in an in-ear device worn by the user. In another embodiment, the enhanced sound signal is provided to a speaker assembly in a head-mounted viewer.

[0107] In some embodiments, the audio system provides one or more filters to the in-ear device. The in-ear device generates an enhanced sound signal by applying the one or more filters to the sound signal.

[0108] The processing and computation performed by the audio system (e.g., target sound source identification, generation of one or more filters applied to the sound emitted by the target sound source, etc.) allows for better enhancement of the audio content presented to the user. For example, a user with hearing loss benefits from an audio controller 230 that identifies a target sound source (emitting sounds at frequencies that the user cannot hear) and generates one or more filters to apply to the sound emitted by the target sound source, which enhance (amplify) those specific frequencies in the sound. Sound sources located in localized areas are better tracked by the microphone array 210 of the head-mounted viewer, which allows for better identification of the target sound source. Moreover, the processing and computation performed by the audio system allows the in-ear device to present more personalized and enhanced audio content to the user when compared to current hearing aids.

[0109] Figure 5 A system 500 comprising a head-mounted viewer 505 and one or more in-ear devices 580 according to one or more embodiments. In some embodiments, the head-mounted viewer 505 may be... Figure 1A Head-mounted viewer 100 or Figure 1B The head-mounted viewer 105 is part of the system 500, which can operate in an artificial reality environment. Figure 5 The illustrated system 500 includes a head-mounted viewer 505, an input / output (I / O) interface 510 coupled to a console 515, a network 520, a map building server 525, and an in-ear device 580. Although Figure 5 The example system 500 shown includes a head-mounted viewer 505, an I / O interface 510, and an in-ear device 580. However, in other embodiments, system 500 may include any number of these components. For example, two in-ear devices may be present, communicating with an associated head-mounted viewer. In another example, multiple head-mounted viewers may be present, each having an associated I / O interface 510, wherein each head-mounted viewer and I / O interface 510 communicates with a console 515. In alternative configurations, system 500 may include different and / or additional components. Additionally, in some embodiments, [the system may be combined with...] Figure 5 The functions described by one or more of the multiple components shown can be combined with Figure 5 The different ways in which these components are described are distributed. For example, some or all of the functions of the console 515 can be provided by the head-mounted viewer 505.

[0110] The head-mounted viewer 505 includes a display assembly 530, an optical module 535, one or more position sensors 540, a DCA 545, a PCA 550, an eye-tracking system 555, and an audio system 560. Some embodiments of the head-mounted viewer 505 have [a combination of...]. Figure 5 The components described are different from those components. Additionally, in combination... Figure 5 The functions provided by the various components described may be distributed in different ways among the various components of the head-mounted viewer 505 in other embodiments, or may be embodied in separate components remote from the head-mounted viewer 505.

[0111] Display component 530 displays content to the user based on data received from console 515. Display component 530 uses one or more display elements (e.g., display element 120) to display the content. The display element may be, for example, an electronic display. In various embodiments, display component 530 includes a single display element or multiple display elements (e.g., one display for each of the user's eyes). Examples of electronic displays include: liquid crystal display (LCD), organic light emitting diode (OLED) display, active-matrix organic light-emitting diode (AMOLED) display, waveguide display, some other display, or some combination thereof. It should be noted that in some embodiments, the display element may also include some or all of the functions of optical module 535.

[0112] Optical module 535 can amplify the received image light from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or both eye-tracking ranges of head-mounted viewer 505. In various embodiments, optical module 535 includes one or more optical elements. Example optical elements included in optical module 535 include: apertures, Fresnel lenses, convex lenses, concave lenses, filters, reflective surfaces, or any other suitable optical elements that affect image light. Furthermore, optical module 535 can include combinations of different optical elements. In some embodiments, one or more of the plurality of optical elements in optical module 535 may have one or more coatings, such as a partial reflective coating or an anti-reflective coating.

[0113] Magnifying and focusing the image light via optical module 535 allows the electronic display to be physically smaller, lighter, and consume less power compared to larger displays. Furthermore, magnification increases the field of view of the content displayed on the electronic display. For example, the field of view of the displayed content is such that it is presented using almost the entire user's field of view (e.g., approximately 110 degrees diagonally), and in some cases, the displayed content is presented using the entire user's field of view. Additionally, in some embodiments, the magnification can be adjusted by adding or removing optical elements.

[0114] In some embodiments, the optical module 535 may be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may include spherical aberration, chromatic aberration, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optical module 535 corrects this distortion when it receives image light from the electronic display based on that content.

[0115] Position sensor 540 is an electronic device that generates data representing position information about head-mounted viewer 505. Position sensor 540 may generate one or more measurement signals in response to movement of head-mounted viewer 505. Position sensor 540 is an embodiment of position sensor 190. Examples of position sensor 540 include one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting motion, or some combination thereof. Position sensor 540 may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU rapidly samples multiple measurement signals and calculates an estimated position of head-mounted viewer 505 based on the sampled data. For example, the IMU integrates multiple measurement signals received from the accelerometers over time to estimate a velocity vector, and integrates the velocity vector over time to determine the estimated position of a reference point on head-mounted viewer 505. A reference point is a point that can be used to describe the position of the head-mounted viewer 505. Although a reference point can generally be defined as a point in space, this reference point is actually defined as a point within the head-mounted viewer 505. (The above reference...) Figure 1A Further operation of position sensor 540 is described.

[0116] The DCA 545 generates depth information for a portion of a local area. The DCA includes one or more imaging devices and a DCA controller. The DCA 545 may also include an illuminator. (See above reference.) Figure 1A The operation and structure of DCA 545 are described.

[0117] The PCA 550 generates color image data of a localized region. The PCA may include one or more imaging devices that acquire images of some or all portions of the localized region. In some embodiments, the PCA 550 may acquire images of one or more sound sources (including the user) within the localized region. (See above references.) Figure 1A Further details regarding the operation and structure of the PCA 550 are described.

[0118] Eye-tracking system 555 determines eye-tracking information associated with the eyes of a user wearing head-mounted viewer 505. The eye-tracking information determined by eye-tracking system 555 may include, for example, gaze information. In some embodiments, eye-tracking system 555 determines pupil measurement information, such as pupil size and pupillary reactivity. Gait information and / or pupil measurement information can be used to determine where the user is looking. Eye-tracking system 555 may use one or more imaging devices to track one or both of the user's eyes. (See above references) Figure 1A Further details on the operation of the eye-tracking system are described.

[0119] Audio system 560 generates one or more filters to apply to the sound signal. Audio system 560 is substantially the same as audio system 200 described above. Audio system 560 may include one or more microphones, one or more transducers, and an audio controller. In some embodiments, the audio system may identify a target sound source in a local region based on explicit user feedback and / or implicit user feedback stored in a model of the local region. In some embodiments, the audio system generates one or more filters to apply to the sound signal associated with the target sound source. In some embodiments, the audio system generates one or more filters to apply to the sound signal associated with other sound sources in the local region.

[0120] I / O interface 510 is a device that enables a user to provide information, send action requests to console 515, and receive responses from console 515. The user can provide information to be stored in audio system 560 (e.g., a user's voice profile). An action request is a request to perform a specific action. For example, an action request may be an instruction to start or stop acquiring image or video data, or an instruction to perform a specific action within an application. I / O interface 510 may include one or more input devices. Example input devices include a keyboard, mouse, game controller, or any other suitable device for receiving action requests and transmitting them to console 515. Action requests received by I / O interface 510 are transmitted to console 515, which performs the action corresponding to the action request. In some embodiments, I / O interface 510 includes an IMU that acquires calibration data indicating an estimated position of I / O interface 510 relative to its initial position. In some embodiments, I / O interface 510 may provide haptic feedback to the user based on instructions received from console 515. For example, haptic feedback can be provided when a motion request is received, or the console 515 can send instructions to the I / O interface 510 when the console 515 performs an action, causing the I / O interface 510 to generate haptic feedback.

[0121] The console 515 provides content for processing to the head-mounted viewer 505 based on information received from one or more of the following: DCA 545, head-mounted viewer 505, and I / O interface 510. Figure 5 In the example shown, console 515 includes an app store 565, a tracking module 570, and an engine 575. Some embodiments of console 515 have a combination with... Figure 5 The modules or components described are different modules or components. Similarly, the functions further described below can be arranged in combination with... Figure 5 The different ways in which they are described are distributed among the various components of console 515. In some embodiments, the functions of console 515 discussed herein may be implemented in head-mounted viewer 505 or a remote system.

[0122] Application store 565 stores one or more applications for execution by console 515. An application is a set of instructions that, when executed by a processor, generate content to be presented to a user. The content generated by the application may respond to user input received via motion of head-mounted viewer 505 or I / O interface 510. Examples of applications include game applications, conferencing applications, video playback applications, or other suitable applications.

[0123] The tracking module 570 uses information from one or more position sensors 540 to track the movement of the head-mounted viewer 505. For example, the tracking module 570 determines the position of a reference point of the head-mounted viewer 505 within a model of a local region based on information from the head-mounted viewer 505. Additionally, in some embodiments, the tracking module 570 can use data portions from the position sensors 540 indicating the position of the head-mounted viewer 505 and a representation of the local region from the DCA 545 to predict the future position of the head-mounted viewer 505. The tracking module 570 provides the engine 575 with the estimated position or predicted future position of the head-mounted viewer 505.

[0124] Engine 575 executes the application and receives position information, acceleration information, velocity information, predicted future position, or a combination thereof from tracking module 570 for head-mounted viewer 505. Based on the received information, engine 575 determines the content to be provided to head-mounted viewer 505 for presentation to the user. For example, if the received information indicates that the user has looked to the left, engine 575 generates content for head-mounted viewer 505 that reflects the user's movement in a virtual local area or local area, enhancing that local area with additional content. Additionally, in response to a received action request from I / O interface 510, engine 575 executes an action within the application running on console 515 and provides feedback to the user that the action has been performed. The provided feedback can be visual or auditory feedback via head-mounted viewer 505, or haptic feedback via I / O interface 510.

[0125] Network 520 couples the head-mounted viewer 505 and / or console 515 to the map building server 525. Network 520 may include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, network 520 may include the Internet and mobile phone networks. In one embodiment, network 520 uses standard communication technologies and / or protocols. Therefore, network 520 may include links using a variety of technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, networking protocols used on Network 520 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), and others. Data exchanged via Network 520 can be represented using various technologies and / or formats, including binary image data (e.g., Portable Network Graphics (PNG)), Hypertext Markup Language (HTML), Extensible Markup Language (XML), and others.In addition, traditional encryption techniques can be used to encrypt all or some of the links. These traditional encryption techniques include, for example, Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec).

[0126] Map building server 525 may include a database storing models describing multiple spaces, wherein a location in the model corresponds to the current configuration of a local region of head-mounted viewer 505. Map building server 525 receives information describing at least a portion of the local region and / or location information of the local region from head-mounted viewer 505 via network 520. Based on the received information and / or location information, map building server 525 determines the location in the model associated with the local region of head-mounted viewer 505. Map building server 525 determines (e.g., retrieves) one or more acoustic parameters associated with the local region, partly based on the determined location in the model and any acoustic parameters associated with that determined location. Map building server 525 may send the location of the local region and any acoustic parameter values ​​associated with the local region to head-mounted viewer 505.

[0127] The above description of embodiments has been presented for illustrative purposes and is not intended to be exhaustive, nor is it intended to limit the patent right to the precise form disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the foregoing disclosure.

[0128] Some portions of this description describe embodiments in terms of the algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to effectively communicate the substance of their work to others skilled in the art. Although these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs or equivalent circuits, or microcode, etc. Furthermore, it has been proven that, without loss of generality, the arrangement of these operations is sometimes referred to as a module for convenience. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0129] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0130] The embodiments may also relate to an apparatus for performing the operations described herein. This apparatus may be specifically constructed for the desired purpose, and / or the apparatus may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system mentioned in this specification may include a single processor, or may be an architecture using a multiple-processor design to increase computing power.

[0131] The embodiments may also relate to a product generated by the computational process described herein. Such a product may include information generated from the computational process, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and such a product may include any embodiment of a computer program product or other combinations of data described herein.

[0132] Finally, the language used in this specification has been chosen primarily for readability and guidance purposes, and may not have been chosen to define or limit the patent rights. Therefore, the scope of the patent rights is not limited by this detailed description, but rather by any claims published in the application based on this document. Thus, the disclosed embodiments are intended to illustrate, not limit, the scope of the patent rights, which are set forth in the appended claims.

Claims

1. A method comprising: Sound is captured from a local area using one or more microphones in a head-mounted viewer; Based on the collected sounds, identify one or more sound sources in the local area; Identify the target sound source among the one or more sound sources; A filter is applied to the acquired sound signal associated with the target sound source, wherein the filter is partially based on a sound profile associated with the user of the head-mounted viewer; An enhanced sound signal is generated by applying the filter to the sound signal; and The enhanced sound signal is provided to the speaker assembly of the head-mounted viewer to be presented to the user.

2. The method according to claim 1, further comprising: Based on feedback from the user, the user's voice profile is updated.

3. The method according to claim 2, wherein, The audio profile includes an audiogram.

4. The method according to any one of the preceding claims, wherein, The enhanced sound signal suppresses multiple other sound sources that are different from the target sound source.

5. The method according to any one of the preceding claims, wherein, The enhanced sound signal spatializes the sound signal to distinguish the target sound source from a plurality of other sound sources different from the target sound source.

6. The method according to claim 5, wherein, Spatializing the sound signal includes increasing the spatial interval between the target sound source and at least one of the plurality of other sound sources, such that the spatial interval exceeds a threshold distance.

7. The method according to any one of the preceding claims, wherein, The enhanced sound signal reduces feedback to improve the quality of the target sound source.

8. The method according to claim 7, wherein, The feedback is reduced using a frequency conversion filter that transposes at least one frequency band by a specific amount.

9. The method according to any one of the preceding claims, wherein, The head-mounted viewing device is an eyeglasses device.

10. The method according to any one of the preceding claims, wherein, The sound profile is personalized based on feedback from the user regarding different audio tones presented to the user via the speaker components of the head-mounted viewer.

11. The method according to claim 10, wherein, The feedback is provided via an I / O interface that is communicatively coupled to the head-mounted viewer.

12. The method according to any of the preceding claims, wherein, The filter includes a head-related transfer function (HRTF).

13. The method according to claim 12, wherein, The HRTF is customized for the user based on machine learning.

14. A head-mounted viewing device, comprising: One or more microphones, the one or more microphones being configured to acquire audio input; One or more speakers, the one or more speakers being configured to output audio output; One or more processors; as well as One or more memories, the one or more memories storing instructions that, when executed by the one or more processors, cause the head-mounted viewer to perform the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing instructions that, when executed by one or more processors of a head-mounted viewer, cause the head-mounted viewer to perform the method according to any one of claims 1 to 13.

Images