Wearer identification based on personalized acoustic transfer function

Abstract

A wearable device includes an audio system. In one embodiment, the audio system includes a sensor array that includes a plurality of acoustic sensors. When a user wears the wearable device, the audio system determines an acoustic transfer function for the user based on detected sounds in a local area around the sensor array. Because the acoustic transfer function is based on the size, shape, and density of the user's body (e.g., the user's head), different acoustic transfer functions will be determined for different users. The determined acoustic transfer function is compared to stored acoustic transfer functions of known users to authenticate the user of the wearable device.

Description

[0001] background

[0002] The present disclosure generally relates to dynamic determination of personalized acoustic transfer functions and use of the acoustic transfer functions for user identification and security purposes.

[0003] Wearable devices such as smart glasses, audio headsets, and augmented reality devices can be used to provide personalized experiences and content to their users. It is desirable to identify or authenticate users, for example, to provide personalized content. Additionally, being able to identify who is wearing the device at any given time can prevent unauthorized users from accessing private or sensitive data.

[0004] Overview

[0005] According to the present invention, an audio system is provided, comprising: a sensor array, the sensor array including a plurality of acoustic sensors configured to detect sound transmitted within a local area of ​​the audio system; and a controller, the controller configured to: determine at least one acoustic transfer function based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the audio system; and identify the user based on the at least one acoustic transfer function.

[0006] Preferably, the controller is configured to identify the user based on the at least one acoustic transfer function by comparing the determined at least one acoustic transfer function with at least one stored acoustic transfer function.

[0007] Conveniently, the plurality of acoustic sensors comprises a first acoustic sensor configured to generate a first audio signal from the sound and a second acoustic sensor configured to generate a second audio signal from the sound; and the controller is configured to determine the acoustic transfer function by determining a ratio between the first audio signal and the second audio signal.

[0008] Preferably, the controller is configured to determine a set of acoustic transfer functions, each acoustic transfer function being based on a ratio between two acoustic sensors of the plurality of acoustic sensors, and the controller is configured to identify the user based on the determined set of acoustic transfer functions.

[0009] Conveniently, the plurality of acoustic sensors comprises a pair of acoustic sensors located adjacent each ear of the user, and the acoustic transfer function corresponds to a head related transfer function (HRTF).

[0010] Preferably, the sensor array is located on a head-mounted device.

[0011] Conveniently, the audio system further comprises a proximity sensor, and wherein the controller is further configured to: determine whether the user is wearing the head-mounted device based on input received from the proximity sensor; and determine the acoustic transfer function in response to determining that the user is wearing the head-mounted device.

[0012] Preferably, the controller is further configured to: determine a source location of the detected sound, the source location indicating a location within a local area relative to the position of the audio system; associate the determined at least one acoustic transfer function with the determined source location; based on the determined source location, identify at least one stored acoustic transfer function corresponding to an authorized user; and identify the user based on a comparison of the determined at least one acoustic transfer function and the identified at least one stored acoustic transfer function.

[0013] Conveniently, at least one stored acoustic transfer function is associated with a source localization within a threshold distance of the determined source localization.

[0014] Preferably, the controller is further configured to cause the speaker to emit sound.

[0015] Conveniently, the sound is generated by the user.

[0016] Preferably, the controller is further configured to perform one or more security functions based on a result of the comparison, wherein the one or more security functions include authorizing the user to access personalized data or setting a configuration of the wearable device based on one or more stored user preferences.

[0017] According to another aspect of the present invention, a method is provided, comprising: detecting sound transmitted within a localized area of ​​a head-mounted device using a sensor array, the sensor array comprising a plurality of acoustic sensors; determining at least one acoustic transfer function associated with a user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the head-mounted device; and identifying the user based on the determined acoustic transfer function.

[0018] Preferably, identifying the user based on the determined acoustic transfer function comprises comparing the determined at least one acoustic transfer function with at least one stored acoustic transfer function.

[0019] Conveniently, the plurality of acoustic sensors comprises a first acoustic sensor configured to generate a first audio signal from the sound and a second acoustic sensor configured to generate a second audio signal from the sound; and wherein determining the acoustic transfer function comprises determining a ratio between the first audio signal and the second audio signal.

[0020] Preferably, the method further comprises: determining whether the user is wearing the head-mounted device based on input received from a proximity sensor of the head-mounted device; and determining the acoustic transfer function in response to determining that the user is wearing the head-mounted device.

[0021] Conveniently, the method further comprises performing one or more security functions based on a result of the comparing, wherein the one or more security functions comprises authorizing the user to access personalization data or setting a configuration of the head mounted device based on one or more stored user preferences.

[0022] Preferably, the method further comprises: determining a source localization of the detected sound, the source localization indicating a location within a local area relative to the position of the head mounted device; associating the determined at least one acoustic transfer function with the determined source localization; based on the determined source localization, identifying at least one stored acoustic transfer function corresponding to an authorized user; and identifying the user based on a comparison of the determined acoustic transfer function and the identified at least one stored acoustic transfer function.

[0023] Conveniently, at least one stored acoustic transfer function is associated with a source localization within a threshold distance of the determined source localization.

[0024] According to another aspect of the present invention, a non-transitory computer-readable medium storing instructions is provided, which instructions, when executed by one or more processors, cause the one or more processors to perform operations including: detecting sound transmitted within a local area of ​​a head-mounted device using a sensor array, the sensor array comprising a plurality of acoustic sensors; determining at least one acoustic transfer function associated with a user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the head-mounted device; and identifying the user based on the determined acoustic transfer function.

[0025] The acoustic transfer function represents the relationship between sound from a source location and how the sound is detected, for example, by a sensor array or by a person. For example, the sound perceived at a person's two ears may be different depending on the direction or location of the sound source relative to each ear, and depending on objects in the local area that may reflect the sound or through which the sound may propagate. In a wearable device that includes an audio system, the acoustic transfer function may be used to present audio content to a user, allowing the user to perceive the presented audio content as originating from a specific location within the local area. Because different users have different head geometries that may affect sound in different ways, different acoustic transfer functions may be associated with different users for the purpose of presenting audio content.

[0026] Embodiments relate to an audio system for identifying a user based on a determined acoustic transfer function. In some embodiments, the audio system includes a sensor array including a plurality of acoustic sensors configured to detect sound transmitted within a local area of ​​the audio system. The audio system may also include a controller. The controller is configured to determine an acoustic transfer function based in part on the detected sound, wherein the acoustic transformation function defines a transformation of the sound caused in part by a head of a user of the audio system, and the controller is configured to identify the user based on the acoustic transfer function.

[0027] In some embodiments, a method of identifying a user based on a determined acoustic transfer function is disclosed. The method may include detecting sound transmitted within a localized area of ​​a head-mounted device using a sensor array, the sensor array including a plurality of acoustic sensors. The method may also include determining at least one acoustic transfer function associated with the user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the head-mounted device. The method may also include identifying the user based on the determined acoustic transfer function.

[0028] In some embodiments, a non-transitory computer-readable medium storing instructions is disclosed, which instructions, when executed by one or more processors, cause the one or more processors to perform operations. The operations may include detecting sound transmitted within a localized area of ​​a head-mounted device using a sensor array, the sensor array comprising a plurality of acoustic sensors. The operations may also include determining at least one acoustic transfer function associated with a user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the head-mounted device. The operations may also include identifying the user based on the determined acoustic transfer function. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A is an example showing a wearable head mounted device implemented as an eyewear device according to one or more embodiments.

[0031] Figure 1B is a perspective view of a head mounted device implemented as a head mounted display in accordance with one or more embodiments.

[0032] Figure 2 is a block diagram of an audio system according to one or more embodiments.

[0033] Figure 3 is a diagram showing that sound in a local area may be detected by a sensor array and used to determine an acoustic transfer function according to some embodiments.

[0034] Figure 4is a flow chart illustrating a process of generating and updating an acoustic transfer function of a wearable device including an audio system according to one or more embodiments.

[0035] Figure 5 is a flow chart illustrating a process for authenticating a user of a wearable device that includes an audio system based on one or more determined acoustic transfer functions in accordance with one or more embodiments.

[0036] Figure 6 is a system environment of an eyewear device including an audio system according to one or more embodiments.

[0037] The accompanying drawings depict embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles or benefits of the present disclosure described herein.

[0038] Detailed Description

[0039] The acoustic transfer function represents the relationship between the sound originating from the source positioning and how the sound is detected (e.g., by a sensor array or by a person). For example, the sound reaching the listener's ear from the source positioning may include the sound that propagates directly from the source positioning to the listener's ear, and the sound that reaches the ear indirectly. The indirect sound may include the sound reflected from the listener's head and / or other objects in the local area, and may also include the sound that passes through the listener's head. Indirect sound may be transformed by diffraction or reflection, resulting in certain frequencies being promoted and other frequencies being attenuated. Therefore, how the sound from a specific source positioning will be perceived by the listener may depend on the size, shape and / or density of the listener's head, ears and face. These same principles can also be applied to the sensor array of the wearable device worn by the user. The sensor array may include at least two microphones at different positions, and the at least two microphones are configured to detect the sound in the local area. How the sound from the source positioning is received by each microphone of the sensor array may depend on the positioning of each microphone relative to the source positioning, as well as the shape, size and / or density of the user's head, ears and face, which affects how the sound is reflected from different parts of the user's body and / or passes through the user's body to indirectly reach at least one microphone of the sensor array. A sensor array (or a person wearing a device having a sensor array) may have several associated acoustic transfer functions corresponding to different source locations and / or frequency ranges associated with detected sounds in a local area around the sensor array (or around the person wearing the sensor array).

[0040] Because the geometry of each user's head is unique to a single user, when a wearable device having a sensor array is worn by a first user, the sensor array of the wearable device can be associated with a first set of acoustic transfer functions that is different from a second set of acoustic transfer functions corresponding to when the wearable device is worn by the second user due to different attributes of the heads of the first user and the second user (e.g., different head shapes, sizes, and / or densities). In this way, the determined acoustic transfer function of the sensor array can be used as an indicator to identify which user is currently wearing the wearable device. As used herein, a user's head shape, size, and / or density may also include features on the user's face (e.g., the size and shape of the user's nose) and features associated with the user's ears (e.g., the shape of the ears and / or ear canals).

[0041] The audio system of the wearable device detects sound in order to generate one or more acoustic transfer functions for the user. In one embodiment, the audio system includes a sensor array and a controller, the sensor array including a plurality of acoustic sensors. Each acoustic sensor is configured to detect sound in a local area around the sensor array. In some embodiments, at least some of the plurality of acoustic sensors are coupled to a near-eye display (NED) configured to be worn by the user. When the user wears the wearable device, the acoustic sensors of the sensor array detect sound in a local area around the user. The detected sound may include controlled sound and uncontrolled sound. Uncontrolled sound is sound that is not controlled by the audio system and occurs in a local area (e.g., naturally occurring ambient noise). Controlled sound is sound controlled by the audio system (e.g., emitted by one or more speakers of the audio system) and / or sound prompted by the audio system (e.g., the audio system prompts the user to generate sound).

[0042] Embodiments of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including head-mounted devices connected to a host computer system, independent head-mounted devices, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0043] Eyewear device configuration

[0044] Figure 1A is an example of a wearable device 100 (e.g., a head mounted device) implemented as an eyewear device according to one or more embodiments. The wearable device 100 is a head mounted device having an eyewear device that presents media to a user. In one embodiment, the wearable device 100 may be a near eye display (NED). In some embodiments, the eyewear device is a near eye display (NED). Typically, the wearable device 100 may be worn on the face of a user such that content (e.g., media content) is presented using a display component and / or an audio system. However, the wearable device 100 may also be used such that media content is presented to the user in a different manner. Examples of media content presented by the wearable device 100 include one or more images, videos, audio, or some combination thereof. The wearable device 100 includes a frame and may include, among other components, a display component including one or more display elements 120, a depth camera component (DCA), an audio system, and a sensor device 190. Although Figure 1A Components of wearable device 100 are shown in example locations on wearable device 100, but components may be located elsewhere on wearable device 100, on a peripheral device paired with wearable device 100, or some combination thereof. Similarly, components on wearable device 100 may be located on a peripheral device that is paired with wearable device 100. Figure 1A More or less than shown.

[0045] The frame 110 holds the other components of the wearable device 100. The frame 110 includes a front portion that holds one or more display elements 120 and an end piece (e.g., a temple) that is attached to the user's head. The front portion of the frame 110 bridges the top of the user's nose. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that curls behind the user's ear (e.g., temple tip, earpiece).

[0046] One or more display elements 120 provide light to a user wearing the wearable device 100. As shown, the head mounted device includes a display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light provided to the eyebox of the wearable device 100. The eyebox is a location in the space occupied by the eye when the user wears the wearable device 100. For example, the display element 120 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional light source, one or more line light sources, one or more point light sources, etc.) and one or more waveguides. Light from the light source is coupled inwardly into one or more waveguides, which output light in a manner such that there is pupil replication in the eyebox of the wearable device 100. The inward coupling and / or outward coupling of light from one or more waveguides can be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a reflector, etc.) that scans the light from the light source when the light from the light source is coupled inwardly into one or more waveguides. It should be noted that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a local area around the wearable device 100. The local area is the area around the wearable device 100. For example, the local area may be a room where a user wearing the wearable device 100 is located, or the user wearing the wearable device 100 may be outside and the local area is an external area. In this context, the wearable device 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent so that light from the local area can be combined with light from one or more display elements to produce AR and / or MR content.

[0047] In some embodiments, display elements 120 do not generate image light, but rather lenses transmit light from localized areas to the viewport. For example, one or both of display elements 120 may be non-corrective lenses (non-prescription lenses), or prescription lenses (e.g., single vision lenses, bifocal and trifocal lenses, or progressive lenses) to help correct for imperfections in the user's vision. In some embodiments, display elements 120 may be polarized and / or tinted to protect the user's eyes from sunlight.

[0048] It should be noted that in some embodiments, the display element 120 may include an additional optical block (not shown). The optical block may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that direct light from the display element 120 to the viewing window. The optical block may, for example, correct for aberrations in some or all of the image content, magnify some or all of the image, or some combination thereof.

[0049] The DCA determines depth information of a portion of a local area around the wearable device 100. The DCA includes one or more imaging devices 130 and a DCA controller (not shown). Figure 1A ), and may also include an illuminator 140. In some embodiments, the illuminator 140 illuminates a portion of the local area with light. The light may be, for example, structured light in infrared (IR) (e.g., dot patterns, bars, etc.), IR flash for time of flight, etc. In some embodiments, one or more imaging devices 130 capture an image of a portion of the local area including light from the illuminator 140. As shown in the figure, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, there is no illuminator 140 and at least two imaging devices 130.

[0050] The DCA controller calculates depth information for a portion of the local area using the acquired images and one or more depth determination techniques. The depth determination techniques may be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from illuminator 140), some other technique for determining the depth of the scene, or some combination thereof.

[0051] The sensor device 190 generates one or more measurement signals in response to the movement of the wearable device 100. The sensor device 190 may be located on a portion of the frame 110 of the wearable device 100. The sensor device 190 may include a position sensor, an inertial measurement unit (IMU), or both. Some embodiments of the wearable device 100 may include or may not include the sensor device 190, or may include more than one sensor device 190. In an embodiment where the sensor device 190 includes an IMU, the IMU generates fast calibration data based on the measurement signals from the sensor device 190. Examples of the sensor device 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor for error correction of the IMU, or some combination thereof. The sensor device 190 may be located outside the IMU, inside the IMU, or some combination thereof.

[0052] Based on one or more measurement signals, the sensor device 190 estimates the current position of the wearable device 100 relative to the initial position of the wearable device 100. The estimated position may include the positioning of the wearable device 100, and / or the orientation of the wearable device 100 or the head of the user wearing the wearable device 100, or some combination thereof. The orientation may correspond to the position of each ear relative to the reference point. In some embodiments, the sensor device 190 estimates the current position of the wearable device 100 using depth information and / or absolute position information from the DCA. The sensor device 190 may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the wearable device 100 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate the velocity vector, and integrates the velocity vector over time to determine the estimated position of the reference point on the wearable device 100. Alternatively, the IMU provides sampled measurement signals to the controller 150, which determines fast calibration data. A reference point is a point that can be used to describe the position of the wearable device 100. Although a reference point can generally be defined as a point in space, in practice, a reference point is defined as a point within the wearable device 100.

[0053] In some embodiments, the wearable device 100 can provide simultaneous localization and mapping (SLAM) of the location of the wearable device 100 and updates of models of local areas. For example, the wearable device 100 may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more RGB cameras that are used to capture images of some or all local areas. In some embodiments, some or all imaging devices 130 of the DCA may also be used as PCAs. The images captured by the PCA and the depth information determined by the DCA can be used to determine parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the sensor device 190 tracks the position (e.g., positioning and posture) of the wearable device 100 in the room.

[0054] In some embodiments, the sensor device 190 may include one or more sensors configured to determine one or more parameters associated with the user of the wearable device 100, such as whether the wearable device is currently being worn by the user. For example, the sensor device 190 may include a proximity sensor (e.g., an infrared proximity sensor), a touch sensor, or some combination thereof that can be used to determine whether the user is currently wearing the wearable device 100. The wearable device may perform one or more functions based on a determination that the user is currently wearing the wearable device 100, such as attempting to authenticate the user, providing content to the user, allowing the user to access one or more functions of the wearable device, and the like. For example, the wearable device 100 may, in response to determining that the user is wearing the wearable device, attempt to authenticate the user and log the user into the user account, and in response to determining that the user is no longer wearing the wearable device, log the user out of the user account.

[0055] The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller 150. However, in other embodiments, the audio system may include different and / or additional components. Similarly, in some cases, the functions described with reference to the components of the audio system may be distributed between the components in a manner different from that described herein. For example, some or all functions of the controller may be performed by a remote server.

[0056] The transducer array presents sound to the user. The transducer array includes multiple transducers. The transducer can be a speaker 160 or a tissue transducer 170 (e.g., a bone conduction transducer or a cartilage conduction transducer). Although the speaker 160 is shown outside the frame 110, the speaker 160 can be enclosed in the frame 110. In some embodiments, instead of a separate speaker for each ear, the wearable device 100 includes a speaker array that includes multiple speakers integrated into the frame 110 to improve the directionality of the presented audio content. The tissue transducer 170 is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to generate sound. The number and / or positioning of the transducers can vary depending on the user's ear. Figure 1A Different than shown.

[0057] The audio system detects sound to generate one or more acoustic transfer functions for the user of the wearable device 100. The acoustic transfer function characterizes how the sound is received from a point in space. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. The audio system can then use the one or more acoustic transfer functions to authenticate the user, generate audio content for the user, etc. The audio system of the wearable device 100 includes a sensor array and a controller 150.

[0058] The sensor array detects sounds in a local area of ​​the wearable device 100. The sensor array includes a plurality of acoustic sensors 180. The acoustic sensor 180 collects sounds emitted from one or more sound sources in a local area (e.g., a room). Each acoustic sensor is configured to detect sounds and convert the detected sounds into an electronic format (analog or digital). The acoustic sensor 180 may be an acoustic wave sensor, a microphone, an acoustic transducer, or a similar sensor suitable for detecting sounds.

[0059] In some embodiments, one or more acoustic sensors 180 can be placed in the ear canal of each ear (e.g., acting as binaural microphones). In some embodiments, acoustic sensors 180 can be placed on an external surface of wearable device 100, placed on an internal surface of wearable device 100, separate from wearable device 100 (e.g., part of some other device), or some combination thereof. The number and / or positioning of acoustic sensors 180 can vary depending on the device. Figure 1A For example, the number of acoustic detection positioning can be increased to increase the amount of audio information collected and the sensitivity and / or accuracy of the information. The acoustic detection positioning can be oriented so that the microphone can detect sounds in a wide range of directions around the user wearing the wearable device 100.

[0060] The sensor array detects sounds in a local area around the sensor array. The local area is the environment surrounding the wearable device 100. For example, the local area may be a room where the user wearing the wearable device 100 is located, or the user wearing the wearable device 100 may be outside the room, and the local area is an external area where the sensor array can detect sounds. The detected sound may be an uncontrolled sound or a controlled sound. An uncontrolled sound is a sound that is not controlled by the audio system and occurs in a local area. An example of an uncontrolled sound may be naturally occurring ambient noise. In this configuration, the audio system is able to calibrate the wearable device 100 using the uncontrolled sound detected by the audio system. A controlled sound is a sound controlled by the audio system. An example of a controlled sound may be one or more signals output by an external system such as a speaker, a speaker assembly, a calibration system, or some combination thereof. In some embodiments, the controlled sound may also include a sound prompted by the audio system. For example, the audio system may prompt the user of the wearable device 100 (e.g., by saying a predetermined phrase) to create a sound. Although the wearable device 100 may be calibrated using an uncontrolled sound, in some embodiments, the wearable device 100 may be calibrated using an external system during the calibration process. Each detected sound (both uncontrolled and controlled) may be associated with a frequency, amplitude, duration, or some combination thereof.

[0061] The configuration of the acoustic sensors 180 of the sensor array may vary. Figure 1A 180, but the number of acoustic sensors 180 can be increased or decreased. Increasing the number of acoustic sensors 180 can increase the amount of audio information collected and the sensitivity and / or accuracy of the audio information. Reducing the number of acoustic sensors 180 can reduce the computing power required for the controller 150 to process the collected audio information. In addition, the position of each acoustic sensor 180 of the sensor array can be changed. The position of the acoustic sensor 180 can include a defined position on the user, a defined coordinate on the frame 110, an orientation associated with each acoustic sensor, or a combination thereof. For example, the acoustic sensor 180 can be located on different parts of the user's ear, such as behind the pinna or in the auricle or fossa, or in addition to the acoustic sensor 180 in the ear canal, additional acoustic sensors can also be provided on or around the ear. Having an acoustic sensor positioned next to the user's ear canal can enable the sensor array to collect information about how sound reaches the ear canal. The acoustic sensors 180 on the frame 110 may be positioned along the length of the temples, across the bridge, above or below the display element 120, or some combination thereof. The acoustic sensors 180 may be oriented so that the sensor array can detect sounds in a wide range of directions around the user wearing the wearable device 100.

[0062] The controller 150 processes information from the sensor array describing the sounds detected by the sensor array. The information associated with each detected sound may include the frequency, amplitude and / or duration of the detected sound. In some embodiments, for each detected sound, the controller 150 performs DoA estimation. The DoA estimation is the estimated direction from which the detected sound arrives at the acoustic sensor of the sensor array. If the sound is detected by at least two acoustic sensors in the sensor array, the controller 150 can use the known positional relationship of the acoustic sensors and the acoustic signals from each acoustic sensor to estimate the source positioning of the detected sound, for example, via triangulation or another DoA algorithm. When the number of acoustic sensors detecting the sound increases and / or when the distance between the acoustic sensors detecting the sound increases, the accuracy of the source positioning estimation can increase.

[0063] In some embodiments, the controller 150 fills the audio data set with information. The information may include the detected sound and parameters associated with each detected sound. Example parameters may include frequency, amplitude, duration, DoA estimation, source localization, or some combination thereof. Each audio data set may correspond to a different source localization relative to the wearable device 100, and include one or more sounds with the source localization. The audio data set may be used to determine one or more acoustic transfer functions for the source localization. One or more acoustic transfer functions may be stored in a data set. In an alternative embodiment, each audio data set may correspond to several source localizations relative to the wearable device 100, and include one or more sounds for each source localization. For example, source localizations that are relatively close to each other (e.g., within a threshold distance) may be grouped together. When the sensor array detects a sound, the controller 150 may fill the audio data set with information. When performing DoA estimation or determining source localization for each detected sound, the controller 150 may further fill the audio data set for each detected sound.

[0064] In some embodiments, the controller 150 selects the detected sound for which DoA estimation and / or source localization determination is performed. The controller 150 may select the detected sound based on the parameters associated with each detected sound stored in the audio data set. The controller 150 may evaluate the stored parameters associated with each detected sound and determine whether one or more stored parameters meet the corresponding parameter conditions. For example, if the parameter is higher or lower than a threshold or falls within a target range, the parameter condition may be met. If the parameter condition is met, the controller 150 performs DoA estimation and / or source localization determination on the detected sound. For example, the controller 150 may perform DoA estimation and / or source localization determination on the detected sound with a frequency within a frequency range, an amplitude higher than a threshold amplitude, a duration lower than a threshold duration, other similar deformations, or some combination thereof. The parameter conditions may be set by the user of the audio system based on historical data, based on analysis of information in the audio data set (e.g., evaluating the collected parameter information and setting an average value) or some combination thereof. The controller 150 may create an element in the audio set to store the DoA estimation and / or source localization of the detected sound. In some embodiments, if the data already exists, the controller 150 may update the element in the audio set.

[0065] In some embodiments, controller 150 may receive location information of wearable device 100 from a system external to wearable device 100. The location information may include the location of wearable device 100, the orientation of wearable device 100 or the head of a user wearing wearable device 100, or some combination thereof. The location information may be defined relative to a reference point. The orientation may correspond to the position of each ear relative to the reference point. Examples of systems include imaging components, consoles (e.g., as in Figure 6 ), a simultaneous localization and mapping (SLAM) system, a depth camera assembly, a structured light system, or other suitable system. In some embodiments, the wearable device 100 may include a sensor that can be used for SLAM calculations, which can be performed in whole or in part by the controller 150. The controller 150 can receive position information from the system continuously or at random or specified intervals. In some embodiments, the received position information can be used to determine the source location of the detected sound. For example, in an embodiment where the detected sound includes a controlled sound generated by an external system, the controller 150 can determine the source location based on the known location of the external system and the current location of the wearable device 100.

[0066] Based on the parameters of the detected sound, the controller 150 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function may be an ATF, an HRTF, other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how the sensor array receives sound from a point in space. Specifically, the ATF defines the relationship between the parameters of the sound at the sound source location and the parameters when the sensor array detects the sound. The parameters associated with the sound may include frequency, amplitude, duration, DoA estimation, etc. In some embodiments, at least some of the acoustic sensors of the sensor array are coupled to the wearable device 100 worn by the user. Since the anatomical structure of a person (e.g., ear shape, shoulders, etc.) affects the sound when the sound propagates to the person's ears, the ATF relative to the specific source location of the sensor array may vary from user to user. Therefore, the ATF of the sensor array is personalized for each user wearing the wearable device.

[0067] HRTF characterizes how ears receive sound from points in space. Since human anatomical structures (e.g., ear shape, shoulders, etc.) affect sound when it propagates to human ears, HRTFs positioned relative to a specific source of a person are unique to each ear of the person (and unique to the person). For example, based on a pair of acoustic sensors located near each ear of the user (e.g., within a threshold distance of the entrance of the ear canal (and can be located at the entrance of the ear canal), or in the ear canal of the ear), the controller 150 can generate two HRTFs for the user, one HRTF for each ear. One HRTF or a pair of HRTFs can be used to create audio content that includes sounds that seem to come from a specific point in space. Several HRTFs can be used to create surround sound audio content (e.g., for home entertainment systems, theater speaker systems, immersive environments, etc.), where each HRTF or each pair of HRTFs corresponds to a different point in space, so that the audio content seems to come from several different points in space. In some embodiments, the controller 150 can update a pre-existing acoustic transfer function based on the DoA estimate of each detected sound. In some embodiments, when the position of the wearable device 100 changes within the local area, the controller 150 can generate a new acoustic transfer function or update a pre-existing acoustic transfer function accordingly.

[0068] Figure 1B 1 is a perspective view of a head mounted device 105 implemented as an HMD according to one or more embodiments. In embodiments describing an AR system and / or an MR system, a portion of the front side of the HMD is at least partially transparent in the visible light band (about 380 nm to about 750 nm), and a portion of the HMD between the front side of the HMD and the user's eyes is at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a strap 175. The head mounted device 105 includes many of the same components as described above with reference to FIG. Figure 1A The components described are the same components, but these components are modified to integrate with the HMD form factor. For example, the HMD includes a display component, a DCA, an audio system, and a sensor device 190. Figure 1B An illuminator 140, a plurality of speakers 160, a plurality of imaging devices 130, a plurality of acoustic sensors 180, and a sensor device 190 are shown. Figure 1B The components of the HMD are shown in a particular position on the head mounted device 105, but it should be understood that in other embodiments, the components of the HMD may be arranged differently. For example, Figure 1B The speaker 160 is shown mounted on the band 175. In other embodiments, the speaker 160 can be on the band 175, on the front rigid body 115, on a separate structure so that the speaker 160 is located near or in the ear of the user, or some combination thereof.

[0069] Audio System Overview

[0070] Figure 2 is a block diagram of an audio system 200 according to one or more embodiments. Figure 1A , Figure 1B The audio system in may be an embodiment of the audio system 200. The audio system 200 detects sound to generate one or more acoustic transfer functions for a user, which may be used to authenticate the user and / or generate audio content for the user. Figure 2 In the embodiment of the present invention, the audio system 200 includes a transducer array 210, a sensor array 220, and an audio controller 230. Some embodiments of the audio system 200 have components different from those described herein. Similarly, in some cases, functions may be allocated between components in a manner different from that described herein.

[0071] The transducer array 210 is configured to present audio content. The transducer array 210 includes a plurality of transducers. A transducer is a device that provides audio content. The transducer may be, for example, a speaker (e.g., speaker 160), a tissue transducer (e.g., tissue transducer 170), some other device that provides audio content, or some combination thereof. The tissue transducer may be configured to be used as a bone conduction transducer or a cartilage conduction transducer. The transducer array 210 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, the transducer array 210 may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range, while a moving coil transducer may be used to cover a second part of a frequency range.

[0072] The bone conduction transducer generates sound pressure waves by vibrating bones / tissues in the user's head. The bone conduction transducer can be coupled to a portion of the headset and can be configured to couple to a portion of the user's skull behind the auricle. The bone conduction transducer receives vibration instructions from the audio controller 230 and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate tissue-propagated sound pressure waves that bypass the eardrum and propagate toward the user's cochlea.

[0073] The bone conduction transducer generates sound pressure waves by vibrating one or more portions of the auricular cartilage of the user's ear. The bone conduction transducer can be coupled to a portion of the head mounted device and can be configured to couple to one or more portions of the auricular cartilage of the ear. For example, the bone conduction transducer can be coupled to the back of the auricle of the user's ear. The bone conduction transducer can be located anywhere along the auricular cartilage around the outer ear (e.g., the auricle, the tragus, some other portion of the cartilage, or some combination thereof). Vibrating one or more portions of the auricular cartilage can generate: air-borne sound pressure waves outside the ear canal; sound pressure waves generated by tissue, which cause certain portions of the ear canal to vibrate, thereby generating air-borne sound pressure waves within the ear canal; or some combination thereof. The generated air-borne sound pressure waves propagate along the ear canal toward the eardrum.

[0074] The transducer array 210 generates audio content according to instructions from the audio controller 230. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that sounds from a specific direction and / or target area (e.g., objects and / or virtual objects in a local area). For example, spatialized audio content can make the sound sound like a virtual singer from across the room of the user of the audio system 200. The transducer array 210 can be coupled to a wearable device (e.g., wearable device 100). In an alternative embodiment, the transducer array 210 can be a plurality of speakers separated from the wearable device (e.g., coupled to an external console). In one embodiment, the transducer array 210 can emit a controlled test sound, which is received by the sensor array 220 and can be analyzed to generate an acoustic transfer function for the wearable device 100. Multiple test sounds with varying frequencies, amplitudes, durations, or sequences can be generated by the transducer array 210. In some embodiments, the audio controller 230 can also issue auditory instructions to the user through the transducer array 210 (e.g., prompting the user to create a sound, move in an area, etc.).

[0075] The sensor array 220 detects sounds in a local area around the audio system 200. The sensor array 220 may include a plurality of acoustic sensors, each of which detects air pressure changes of sound waves and converts the detected sound into an electronic format (analog or digital). The plurality of acoustic sensors may be located on a head mounted device (e.g., the head mounted device 100 and / or the head mounted device 105), on a user (e.g., in the ear canal of the user), on a neckband, or some combination thereof. The acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, a contact microphone, or any combination thereof. In some embodiments, the sensor array 220 is configured to monitor the audio content generated by the transducer array 210 using at least some of the plurality of acoustic sensors. Increasing the number of sensors may improve the accuracy of the information (e.g., directionality) describing the sound field generated by the transducer array 210 and / or the sound from the local area.

[0076] As reference Figure 1A , Figure 1B The detected sound may be an uncontrolled sound or a controlled sound. Each detected sound may be associated with audio information (e.g., frequency, amplitude, duration, or some combination thereof). Each acoustic sensor in the sensor array 220 may be active (powered on) or inactive (powered off). The acoustic sensor is activated or deactivated according to an instruction from the audio controller 230. In some embodiments, all acoustic sensors in the sensor array 220 may be active to detect sound, or a subset of multiple acoustic sensors may be active. The active subset includes at least two acoustic sensors among the multiple acoustic sensors. The active subset may include, for example, every other acoustic sensor, a preprogrammed initial subset, a random subset, or some combination thereof.

[0077] The audio controller 230 controls the operation of the audio system 200. Figure 2 In some embodiments, the audio controller 230 includes a data storage 235, a source location determination module 240, a transfer function processing module 250, a tracking module 260, a beamforming module 270, and a sound filter module 280. In some embodiments, the audio controller 230 can be located inside the head-mounted device. Some embodiments of the audio controller 230 have different components than those described herein. Similarly, functions can be allocated between components in a manner different from that described herein. For example, some functions of the controller can be performed external to the head-mounted device (e.g., the learned transfer function storage device stored by the data storage 235 and the security module 285 can be implemented as part of a separate security system that communicates with the audio system 200).

[0078] The data storage 235 stores data for use by the audio system 200. The data in the data storage 235 may include sounds recorded in a local area of ​​the audio system 200, audio content, calculated HRTFs, transfer functions of one or more sensors, ATFs of one or more acoustic sensors, sound source localization, virtual models of the local area, direction of arrival estimates, sound filters, and other data related to use by the audio system 200, or any combination thereof.

[0079] The audio controller 230 processes the information from the sensor array 220. In addition, the audio controller 230 controls other modules and devices of the audio system 200. The information associated with each detected sound may include the frequency, amplitude, and / or duration of the detected sound.

[0080] The source localization determination module 240 is configured to locate the sound source in the local area based in part on the information from the sensor array 220. Localization is the process of determining the position of the sound source relative to the user of the audio system 200. If the sound is detected by at least two sound sensors in the sensor array, the audio controller 230 can use the positional relationship of the sound sensors and the DoA estimation from each sound sensor to estimate the source localization of the detected sound, for example, via triangulation. The estimated source localization can be the relative position of the source localization in the local area relative to the position of the sensor array 220. The DOA analysis can include analyzing the intensity, spectrum and / or arrival time of each sound at the sensor array 220 to determine the direction from which the sound originates. In some cases, the DOA analysis can include any suitable algorithm for analyzing the surrounding acoustic environment in which the audio system 200 is located.

[0081] For example, the DOA analysis may be designed to receive an input signal from the sensor array 220 and apply a digital signal processing algorithm to the input signal to estimate the direction of arrival. These algorithms may include, for example, a delay and sum algorithm in which the input signal is sampled and the resulting weighted and delayed versions of the sampled signal are averaged together to determine the DOA. A least mean square (LMS) algorithm may also be implemented to create an adaptive filter. The adaptive filter may then be used, for example, to identify differences in signal strength or differences in arrival time. These differences may then be used to estimate the DOA. In another embodiment, the DOA may be determined by converting the input signal into the frequency domain and selecting a specific bin in the time-frequency (TF) domain to be processed. Each selected TF bin may be processed to determine whether the bin includes a portion of the audio spectrum having a direct path audio signal. Those bins having a portion of the direct path signal may then be analyzed to identify the angle at which the sensor array 220 receives the direct path audio signal. The determined angle may then be used to identify the DOA of the received input signal. Other algorithms not listed above may also be used alone or in combination with the above algorithms to determine the DOA.

[0082] In some embodiments, if the absolute position of the sensor array 220 is known in the local area, the audio controller 230 can determine the absolute position of the source positioning. The position of the sensor array 220 can be received from an external system (e.g., an imaging component, an AR or VR console, a SLAM system, a depth camera component, a structured light system, etc.). The external system can create a virtual model of the local area, in which the local area and the position of the sensor array 220 are mapped. The received position information may include the positioning and / or orientation of the sensor array in the mapped local area. The audio controller 230 can update the mapping of the local area with the determined source positioning of the detected sound. The audio controller 230 can receive position information from an external system continuously or at random or specified intervals. In some embodiments, the audio controller 230 selects the detected sound (e.g., only the sound that satisfies certain parameter conditions) for which the source positioning determination is performed.

[0083] In some embodiments, the source location determination module 240 can receive information from one or more additional sensors (e.g., sensor device 190) for use in determining the source location of the detected sound. For example, in some embodiments, one or more camera assemblies, gyroscopes, accelerometers, magnetometers, SLAM systems, and / or the like can be used to determine the ground truth position and orientation of the wearable device, from which the relative position of one or more controlled sound sources (e.g., external speakers) can be determined. For example, in the case where the detected sound is a controlled sound emitted from an external speaker at a known location relative to a reference point, the source location determination module 240 uses the position of the audio system 200 relative to the reference point to determine the source location of the sound relative to the audio system 200. In some embodiments, if the wearable device is moving when the sound is detected (e.g., due to the user moving in a local area), the additional sensor data can be used to compensate for the movement when determining the source location of the detected sound.

[0084] The transfer function processing module 250 is configured to generate one or more acoustic transfer functions. Generally speaking, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function processing module 250 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function can be an ATF, HRTF, other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how a microphone receives sound from a point in space. In Figure 2 In an embodiment of the present invention, the acoustic transfer function represents the relationship between the sound at the sound source location and how the sound is detected, for example, by a sensor array or by a person. When sounds originating from different source locations are detected, the transfer function processing module 250 can determine one or more acoustic transfer functions for each different source location.

[0085] In one embodiment, the transfer function processing module 250 generates an array transfer function (ATF). The ATF characterizes how the sensor array 220 receives sound from a point in space. Specifically, the ATF defines the relationship between the parameters of the sound at the sound source location and the parameters when the sensor array 220 detects the sound. The parameters associated with the sound may include frequency, amplitude, duration, etc. The transfer function processing module 250 may generate one or more ATFs for a specific source location of the detected sound. Factors that may affect how the sound is received by the sensor array 220 may include the arrangement and / or orientation of the acoustic sensors in the sensor array 220, any objects between the sound source and the sensor array 220, the anatomical structure of the user wearing the eyewear device with the sensor array 220, or other objects in the local area. For example, if a user is wearing an eyewear device including the sensor array 220, the anatomical structure of the person (e.g., ear shape, shoulders, etc.) may affect the sound waves when they propagate to the sensor array 220. In another example, if the user is wearing an eyewear device including the sensor array 220 and the local area surrounding the sensor array 220 is an external environment including buildings, trees, bushes, a body of water, etc., these objects may attenuate or amplify the amplitude of the sound in the local area. Generating and / or updating the ATF may improve the accuracy of the audio information collected by the sensor array 220.

[0086] In one embodiment, the transfer function processing module 250 generates one or more HRTFs. HRTFs characterize how a person's ears receive sound from a point in space. Because a person's anatomical structure (e.g., ear shape, shoulders, etc.) affects sound as it propagates to a person's ears, an HRTF relative to a person's specific source positioning is unique to each ear of the person (and unique to that person). The transfer function processing module 250 can generate multiple HRTFs for a single person, where each HRTF can be associated with a different source positioning, a different frequency range, or some combination thereof.

[0087] The security module 285 is configured to manage one or more security functions related to user identification or authentication based on the determined acoustic transfer function using the acoustic transfer function generated for the user by the transfer function processing module 250. The one or more security functions may include identifying or authenticating a user based on their determined acoustic transfer function, based on a comparison to a previously learned acoustic transfer function, logging a user into a user profile or blocking a user from logging into a user profile based on the identification or authentication, allowing or blocking access to personalized content based on the identification or authentication, and the like. To enable identification of the user, the determined acoustic transfer function associated with the user is compared to previously determined acoustic transfer functions associated with known users (hereinafter referred to as "personalized functions") stored in the data storage 235. For example, in some embodiments, the security module 285 retrieves one or more personalized functions of one or more known users from the data storage 235 in response to receiving the function determined for the user, and attempts to identify the user based on the comparison of the determined acoustic transfer function with the retrieved personalized functions. In some embodiments, the security module 285 retrieves only personalized functions corresponding to source locations associated with the determined acoustic transfer function, or personalized functions associated with source locations within a threshold distance of the determined source location of the acoustic transfer function. In some embodiments, instead of or in addition to retrieving from the data storage 235, the security module 285 may retrieve personalized functions associated with one or more known users from a transfer function storage device located on a computer associated with the wearable device (e.g., Figure 6 console 615 as shown) or a server accessible via a network (e.g., Figure 6 In addition, although the security module 285 is shown as part of the audio system 200, in some embodiments, the functions of the security module 285 can be performed on a separate device (e.g., a console) or server, where the remote security module receives the acoustic transfer function determined at the audio system 200 and performs one or more security functions with respect to the acoustic transfer function.

[0088] The security module 285 may use a regression model, a clustering model, or a machine learning based classification model to compare the determined acoustic transfer function and the retrieved personalized function. For example, in some embodiments, the security module 285 includes a neural network trained to determine similarity levels between different acoustic transfer functions.

[0089] In some embodiments, the security module 285 determines a confidence level indicating a level of similarity between one or more acoustic transfer functions determined for the user and one or more stored personalized functions corresponding to known users, and authenticates the user as a known user if the confidence level exceeds a threshold. In some embodiments, the confidence level may be based on the number of acoustic transfer functions compared, the number of different positioning associated with the compared functions, etc. In some embodiments, if the confidence level is within a predetermined range, the security module 285 may prompt the user to provide additional authentication information (e.g., a password, fingerprint, voice recognition, etc.). In some embodiments, the confidence value may be aggregated with other security measures to form an aggregated confidence metric for identifying the user.

[0090] The data storage 235 may store (e.g., by the transfer function processing module 250) the acoustic transfer functions determined for known users as personalized functions for the users. In some embodiments, the data storage 235 stores one or more sets of personalized functions for each of the one or more known users, each set of personalized functions being associated with a particular source location. Each source location may correspond to a location relative to the audio system 200, or to a plurality of locations within a particular area (e.g., a set of locations within a threshold distance of each other). In some embodiments, each set of personalized functions may be further divided into one or more subsets corresponding to different frequency ranges. In some embodiments, the personalized functions determined for one or more users may be stored in a separate learned transfer function storage device (not shown).

[0091] In some embodiments, the security module 285 performs an initial calibration operation for the user in order to populate the data store 235 with personalized functions associated with the user so that when the user later wears the wearable device, their personalized functions can be used to identify the user. During the calibration operation, the user is authenticated and associated with a known user profile. For example, the user may enter a password on a user interface associated with the device, provide a fingerprint, and / or be authenticated using other means. When the user wears the wearable device, the transfer function processing module 250 determines one or more acoustic transfer functions for the user, which are stored in the data store 235 as the personalized functions for the user. For example, in some embodiments, after the user is first authenticated via some other means (e.g., a fingerprint on a puck, phone, or other trusted device), the device is trained for the user. When the device is worn (e.g., as determined by an infrared (IR) proximity sensor or other don / doff sensor on the inside of an eyeglass frame), the acoustic transfer function of the wearer is predicted / estimated during a "device-on" state.

[0092] In some embodiments, the security module 285 may cause one or more controlled sounds to be emitted in order to determine an acoustic transfer function corresponding to certain desired positioning and / or frequency ranges for the user. For example, the security module 285 may cause one or more speakers of the transducer array 210 to emit one or more predetermined sounds. In other embodiments, the security module 285 may cause a speaker of an external device that communicates with the wearable device 100 to emit a predetermined sound. The security module 285 may prompt the user to create one or more sounds, such as saying one or more phrases, clapping, etc. In some embodiments, the security module 285 may prompt the user to move within a local area in order to detect sounds originating from a greater variety of different positionings relative to the device. The calibration operation may be performed until a threshold number of acoustic transfer functions are determined for the user, until an acoustic transfer function corresponding to certain source positioning, a threshold number of source positioning, and / or certain frequency ranges is determined for the user, and so on. In some embodiments, as the calibration operation continues to determine and / or update the acoustic transfer function for the user, until a threshold confidence level as determined by the machine learning model is reached.

[0093] Once the calibration operation has been performed and the personalized function is stored for the user, the user can be identified based on the personalized function of the user when the user wears the device later. For example, when it is determined that the user is wearing the wearable device, the transfer function processing module 250 dynamically determines one or more acoustic transfer functions for the user. In some embodiments, the transfer function processing module 250 uses uncontrolled sounds in the local area of ​​the user to determine the acoustic transfer function for the user. In some embodiments, the security module 285 can cause the emission of controlled sounds, or can prompt the user (e.g., by saying a predetermined phrase) to create one or more sounds that can be used to determine the acoustic transfer function. The use of controlled sounds can allow the acoustic transfer function associated with a specific source location or a specific frequency range to be reliably determined, regardless of the level of uncontrolled sounds in the local area. In addition, in an embodiment where the user is prompted to create one or more sounds, the determination of the user's acoustic transfer function can be performed in parallel with the user's voice recognition. For example, determination of the acoustic transfer function may utilize the user's spatial signature when creating sounds (e.g., because the sounds originate from consistent, predictable positioning relative to the audio sensor), while speech recognition functionality may analyze the user's speech signature based on the created sounds.

[0094] The determined acoustic transfer function may then be compared to at least a subset of the user's personalized functions stored in data storage 235, wherein the user may be identified in response to a successful comparison (e.g., a confidence value exceeding a threshold level) and the user may be allowed access to personalized content and / or certain features of the wearable device. In another example, a user-specific configuration of the wearable device may be activated in response to identifying the user. On the other hand, if the comparison is unsuccessful (e.g., due to too large a difference between the determined acoustic transfer function and the stored personalized function), the user is not identified. In some embodiments, the user may be restricted from accessing certain data or features of the wearable device. In some embodiments, the results of the comparison may be used in conjunction with one or more additional identification methods (e.g., voice recognition, eye / iris recognition, fingerprint recognition, etc.) to identify the user (e.g., a weighted combination of confidence values ​​associated with multiple different identification schemes).

[0095] In some embodiments, the acoustic transfer function can be used to facilitate user authentication. For example, the stored personalized function of the user can be used as a "fingerprint" of the user, similar to an actual fingerprint, face ID system, etc. The security module 285 can log the first user into the first user account when the first user is identified and authenticated. Later, the security module 285 can determine that a different wearer is wearing the wearable device (e.g., by detecting that the wearable device is removed and then re-worn, or by periodically determining the acoustic transfer function of the wearing user), and in response to determining that the current wearing user is different from the first user based on a comparison of the acoustic transfer function determined for the user and the stored personalized function of the first user, log out of the first user account. In some embodiments, if the first user is identified as being part of an authorized group associated with the second user (e.g., a family member of the second user), the security module 285 can log the first user into a user account or profile associated with the second user.

[0096] The beamforming module 270 is configured to process one or more ATFs to selectively emphasize the sound from the sound source in a certain area while de-emphasize the sound from other areas. When analyzing the sound detected by the sensor array 220, the beamforming module 270 can combine information from different sound sensors to emphasize the associated sound from a specific area of ​​the local area while de-emphasize the sound from the outside of the specific area. The beamforming module 270 can isolate the audio signal associated with the sound from a specific sound source from other sound sources in the local area based on different DOA estimates, such as from the source positioning determination module 240 and the tracking module 260. The beamforming module 270 can therefore selectively analyze the discrete sound sources in the local area. In some embodiments, the beamforming module 270 can enhance the signal from the sound source. For example, the beamforming module 270 can apply a sound filter that eliminates signals above, below, or between certain frequencies. Signal enhancement works to enhance the sound associated with a given identified sound source relative to other sounds detected by the sensor array 220. In some embodiments, beamforming module 270 includes one or more beamforming filters adapted to contain a clean copy of the ATF for one or more given directions and / or environments.

[0097] The sound filter module 280 determines the sound filter for the transducer array 210. In some embodiments, the sound filter spatializes the audio content so that the audio content sounds like it originates from the target area. The sound filter module 280 can use HRTF and / or acoustic parameters to generate the sound filter. The acoustic parameters describe the acoustic characteristics of the local area. The acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the sound filter module 280 calculates one or more acoustic parameters. In some embodiments, the sound filter module 280 receives the sound filter from a mapping server (e.g., as described below with reference to Figure 6 The sound filter module 280 provides a sound filter to the transducer array 210. In some embodiments, the sound filter can cause positive or negative amplification of the sound as a function of frequency.

[0098] Determination of acoustic transfer functions for user authentication

[0099] Figure 3 is a diagram showing that sounds in a local area can be detected by a sensor array and used to determine an acoustic transfer function according to some embodiments. A user can wear a wearable device having acoustic sensors 305A and 305B (collectively referred to as acoustic sensors 305) near their head 310. Although Figure 3Only two acoustic sensors 305 are shown in a particular position relative to the user's head 310, but it should be understood that in other embodiments, the audio system of the wearable device may include additional acoustic sensors, acoustic sensors in different positions relative to the user's head 310, or some combination thereof.

[0100] Sound originating from the source localization 315 in the local area may generate sound waves that may be detected by the acoustic sensor 305. The sound detected at each of the acoustic sensors 305A and 305B may include direct sound 320A and 320B, respectively, and indirect sound. The indirect sound may include reflected sound 325A, 325B reflected from the user's head 310, and sound 330 propagated through at least a portion of the user's head 310. In some embodiments, the direct sound may include sound propagated through different media (e.g., the user's head 310). In some embodiments, the indirect sound may also include sound reflected from other objects in the local area (not shown). In some embodiments, since the direct sound has the most direct path from the source localization 315 to each acoustic sensor 305 (e.g., typically, a line of sight path, unless blocked), the acoustic sensor will detect the direct sound earliest in time, while the indirect sound may be detected later in time. Since the user's head 310 is close to the sound sensor 305, the indirect sound corresponding to the reflection from the user's head can closely interact with the direct sound, while the sound corresponding to the reflection from other objects in the local area (e.g., walls, furniture, etc.) can be detected at the sound sensor 305 later in time.

[0101] The audio system determines an acoustic transfer function based on the sound detected at each acoustic sensor 305. Due to the different positioning of the acoustic sensors 305, the direct sound 320A and 320B are detected by the acoustic sensors 305A and 305B at different times. In addition, based on the properties of any intervening objects between the source positioning 315 and each acoustic sensor 305 (e.g., the shape, size, and / or density of the user's head 310), the amplitude of the direct sound 320 detected at each acoustic sensor 305 may be different. Based on the position of the user's head 310 relative to the source positioning 315, the geometry and density of the user's head 310, etc., the indirect sound (including reflected sound 325A, 325B and sound 330 passing through the head) detected at each acoustic sensor 305A and 305B may also be different. Therefore, for a given sound originating from the same source positioning 315, the difference between the direct sound and the indirect sound detected at each acoustic sensor 305 will be different for different users with different head shapes, thereby allowing different acoustic transfer functions to be determined for different users.

[0102] In some embodiments, the determined acoustic transfer function is a relative transfer function that corresponds to the relative difference between sounds detected at two or more microphones. Because the relative transfer function reflects the relative difference between sounds detected at different microphones, sounds with different auditory content from the same location can be used to determine the relative transfer function of the user. This allows the user to be authenticated by comparing a first transfer function determined for the user based on a first sound originating from a known source location with a second transfer function determined for the user based on a second sound originating from the source location, even if the first sound and the second sound are different.

[0103] In embodiments where the sensor array of the audio system includes more than two acoustic sensors, a predetermined acoustic sensor may be designated as a reference acoustic sensor, wherein the determined relative transfer function indicates a relative difference between sounds detected at each remaining acoustic sensor and the reference acoustic sensor. In some embodiments, the audio system determines a set of acoustic transfer functions, each acoustic transfer function corresponding to a ratio between sounds detected at two acoustic sensors of the sensor array.

[0104] In some embodiments, the audio system analyzes the sound received at each acoustic sensor 305 and extracts a portion of the sound from the total sound detected at each acoustic sensor. The extracted portion of the sound may correspond to the sound received from the source location within the time window in which the first portion of the direct sound from the source location is received. Therefore, the extracted direct sound may include the sound (e.g., sound 320) directly propagated through the air to each acoustic sensor, the direct sound propagated through other media (e.g., the user's head 310), and / or the indirect sound reflected from part of the user's head. On the other hand, the time window may be selected so that the extracted sound will not contain the indirect sound reflected from a more distant object (e.g., other parts of the user's body, other objects in the local area). In some embodiments, the direct sound is identified as arriving at each acoustic sensor 305 earlier and having a larger amplitude relative to the indirect sound. The relative transfer function is determined based on the difference between the extracted sounds detected at each acoustic sensor, which reflects the relative positioning of the acoustic sensor 305 to the source location 315, and any influence from the user's head 310. In some embodiments, the difference between the extracted sounds detected at each acoustic sensor is determined as a ratio. Because only the extracted sound detected by the acoustic sensor is used to determine the relative transfer function, the impact of indirect sound due to reflections from objects in the local area other than the user's head 310 is ignored and does not affect the determined acoustic transfer function. Therefore, based on the position of the audio system relative to the source positioning, the impact on the direct sound due to the user's head 310, and the frequency range of the detected sound, the determined acoustic transfer function will reflect the time delay and frequency response of the sound detected at the acoustic sensor 305. Additional examples of methods for extracting direct sound from received sound and determining an acoustic transfer function are described in U.S. Patent Application No. 16 / 015,879, entitled "Audio System for Dynamic Determination of Personalize Acoustic Transfer Functions," which is incorporated herein by reference in its entirety.

[0105] Acoustic Transfer Function Certification Process Flow

[0106] Figure 4 4 is a flow chart illustrating a process 400 for generating and updating an acoustic transfer function of a wearable device (e.g., wearable device 100) including an audio system (e.g., audio system 200) according to one or more embodiments. In one embodiment, Figure 4The process is performed by components of the audio system. In other embodiments, other entities (e.g., a console) may perform some or all steps of the process. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0107] The audio system monitors 410 sounds in a local area around a sensor array on the eyewear device. The sensor array can detect sounds, such as uncontrolled sounds and controlled sounds present in the local area. Each detected sound can be associated with a frequency, an amplitude, a duration, or some combination thereof. In some embodiments, the audio system stores information associated with each detected sound in an audio data set.

[0108] In some embodiments, the audio system optionally estimates 420 the position of the sensor array in the local area. The estimated position may include the positioning of the sensor array, and / or the orientation of the head of the wearable device or the user wearing the wearable device, or some combination thereof. In one embodiment, the audio system may include one or more sensors that generate one or more measurement signals in response to the movement of the sensor array. The audio system may estimate the current position of the sensor array relative to the initial position of the sensor array. In another embodiment, the audio system may receive position information of the eyewear device from an external system (e.g., an imaging component, an AR or VR console, a SLAM system, a depth camera component, a structured light system, etc.). In some embodiments, knowledge of the positioning of the device relative to the environment can be used to help remove audio data corresponding to reflections from objects in the environment, thereby allowing the calculated acoustic transfer function to more accurately reflect the geometry of the user's head by reducing the impact of reflected sounds from other objects in the environment.

[0109] The audio system performs 430 source localization determination for each detected sound relative to the position of the sensor array. In some embodiments, the source localization determination is based on one or more DoA estimates for each acoustic sensor of the microphone, the DoA estimate indicating an estimated direction of arrival of the detected sound at the acoustic sensor of the sensor array. The DoA estimate can be represented as a vector between the estimated source localization of the detected sound and the position of the eyewear device within the local area. The source localization can be determined based on triangulation of the estimated DoA. In some embodiments, the audio system can perform 430 source localization determination for the detected sound associated with a parameter that satisfies the parameter condition. For example, the parameter condition can be satisfied if the parameter is above or below a threshold or falls within a target range.

[0110] The audio system updates 440 one or more acoustic transfer functions. The acoustic transfer function may be an ATF or an HRTF. The acoustic transfer function represents the relationship between the sound at the sound source location and how the sound is detected. Thus, each acoustic transfer function is associated with a different source location (relative to the location of the sensor array) of the detected sound. As a result, the audio system may update 440 multiple acoustic transfer functions for a particular source location.

[0111] In some embodiments, the eyewear device may update 440 two HRTFs for a particular source location, one HRTF for each ear of the user.In some embodiments, the audio system generates one or more acoustic transfer functions, each associated with a relative ratio of sounds detected at different acoustic sensors of the sensor array.

[0112] When different sounds are detected from the local area, the audio system can generate one or more new acoustic transfer functions, or correspondingly update 440 one or more pre-existing acoustic transfer functions associated with source localization. Process 400 can be continually repeated as a user wearing the sensor array moves through the local area, or process 400 can be initiated when a sound is detected via the sensor array.

[0113] Figure 5 5 is a flow chart illustrating a process 500 of identifying a user of a wearable device including an audio system based on one or more determined acoustic transfer functions according to one or more embodiments. The audio system receives 510 an indication that the user is wearing the wearable device. In some embodiments, the indication can be based on an operating state of the wearable device (e.g., the wearable device is turned on, one or more functions are turned on, etc.), one or more sensor readings (e.g., a proximity sensor indicating that the user is wearing the device, such as an IR proximity sensor), one or more user inputs (e.g., the user presses a particular button after wearing the wearable device), or some combination thereof.

[0114] As discussed above, while it is possible to determine the acoustic transfer function of a user based on sounds in a local area without requiring any input from the user, in some embodiments, the audio system can cause 520 to emit controlled sounds and / or prompt the user to generate one or more sounds in order to facilitate determining the acoustic transfer function for the user. For example, the audio system can cause a speaker controlled by or in communication with the wearable device to produce one or more sounds. In some embodiments, the user can be prompted to say one or more predetermined phrases, clap, etc. In embodiments where it is known that the personalized function of the user includes an acoustic transfer function calculated based on phrases spoken by the user during a calibration operation, prompting the user to say one or more phrases can ensure that an acoustic transfer function corresponding to a specific source location can be obtained, because the user's mouth position will typically be at a similar location when the user wears the device.

[0115] Based on the sounds detected at the sensor array (e.g., sensor array) of the audio system, the audio system determines 530 at least one acoustic transfer function for the user. For example, the acoustic sensors of the sensor array may be located on a headset worn on the user's head. Sound is propagated from the source location to each acoustic sensor. The acoustic sensor generates audio data based on the collected sound. The sound is transformed by reflecting from different parts of the user's head (e.g., head, face, ear surface) and / or propagating through the user's head, and this transformation may be different for different users. The transformation may include frequency response, time delay, and / or amplitude change. The at least one acoustic transfer function may be determined based on the ratio of audio data collected by the acoustic sensors at different locations. In some embodiments, a set of acoustic transfer functions may be determined. For example, the ratio of audio data between pairs of acoustic sensors and for each discrete source location may be used to determine the set of acoustic transfer functions.

[0116] In some embodiments, the sound can be a controlled sound (e.g., a sound emitted by a speaker controlled by or in communication with the wearable device, or a sound spoken by the user in response to a prompt). In other embodiments, the detected sound can include uncontrolled sound in a local area of ​​the device. In some embodiments, the audio system uses Figure 4 The process shown determines at least one acoustic transfer function. The determined acoustic transfer function may include a relative transfer function indicating relative differences in the sounds detected by each microphone of the sensor array. In some embodiments, each determined acoustic transfer function is associated with a specific source location, a specific frequency range, or a combination thereof.

[0117] The audio system compares 540 at least one acoustic transfer function determined for the user with one or more stored acoustic transfer functions associated with known users. In some embodiments, the audio system identifies one or more stored personalized functions associated with a location that is the same as or within a threshold distance of a source location associated with at least one determined acoustic transfer function for comparison with the determined acoustic transfer function. In some embodiments, the identified personalized function may be determined as a function associated with a common frequency range with the determined acoustic transfer function. In some embodiments, the audio system determines one or more confidence values ​​based on a comparison between at least one determined acoustic transfer function and the identified stored acoustic transfer function associated with each known user. In some embodiments, a set of acoustic transfer functions (e.g., each acoustic transfer function corresponding to a ratio generated using a different pair of acoustic sensors) may be compared with a corresponding set of stored acoustic transfer functions. In some embodiments, the comparison of the array transfer function with the set of stored acoustic transfer functions may be accomplished using machine learning and statistical learning algorithms, including similarity learning, representation learning, and distance metric calculations. In some embodiments, such specific models / algorithms include kernalized nearest neighbors, neural networks, nonlinear transformations (such as single index or multi-index models), generalized principal component analysis, local linear embedding, isometric mapping, etc.

[0118] The audio system identifies 550 the user based on the comparison of the acoustic transfer function. For example, if the confidence obtained by comparing the determined one or more acoustic transfer functions with the stored acoustic transfer function of the first known user meets the threshold, the user can be identified as the first known user. In some embodiments, the comparison result can be combined with one or more additional authentication methods (e.g., voice, fingerprint, etc.) to identify or authenticate the user. For example, if the user is prompted to say a predetermined phrase (e.g., at 520), the sound corresponding to the user's voice can be used to determine the user's acoustic transfer function, and perform voice recognition on the user. In some embodiments, if the user is identified as a known user, the device can perform one or more actions personalized to the user, for example, automatically configuring one or more settings according to one or more stored user preferences, allowing the user to access personalized content, etc. On the other hand, if the user cannot be identified as a known user, the device can prevent the user from accessing the personalized content of the known user, and / or limit one or more functions. In some embodiments, the audio system can prompt authorized users to create a user profile (e.g., undergo a calibration operation) so that they can become known users of the device.

[0119] By using the audio system of the wearable device to dynamically determine the acoustic transfer function of the user wearing the wearable device, the user can be identified as a known user of the device by comparing the determined acoustic transfer function with previously determined and stored acoustic transfer functions associated with known users. This allows for user identification and / or authentication in a hands-free manner that is unobtrusive to the user (especially if uncontrolled sound in a local area is used, so no action is required by the user) and / or can be easily cross-referenced with other authentication methods (e.g., voice recognition) for additional security. In addition, the determined acoustic transfer function can also be used to provide audio content to the user during operation of the wearable device.

[0120] Example system environment

[0121] Figure 6 600 includes a head mounted device 605 according to one or more embodiments. In some embodiments, the head mounted device 605 may be Figure 1A Wearable device 100 or Figure 1B The head mounted device 105. The system 600 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 6 The system 600 shown includes a head mounted device 605, an input / output (I / O) interface 610 coupled to a console 615, a network 620, and a mapping server 625. Figure 6 An example system 600 is shown that includes one head mounted device 605 and one I / O interface 610, but in other embodiments, any number of these components may be included in the system 600. For example, there may be multiple head mounted devices, each with an associated I / O interface 610, each head mounted device and I / O interface 610 communicating with a console 615. In alternative configurations, different and / or additional components may be included in the system 600. Additionally, in some embodiments, in combination with Figure 6 The functionality described by one or more components shown may be different in combination. Figure 6 The described approach is distributed between components. For example, some or all of the functions of console 615 may be provided by head mounted device 605.

[0122] The head mounted device 605 includes a display assembly 630, an optical block 635, one or more position sensors 640, and a DCA 645. Some embodiments of the head mounted device 605 have a Figure 6 In addition, in other embodiments, by combining Figure 6 The functionality provided by the various components described may be distributed differently among components of the head mount 605 , or collected in a separate component remote from the head mount 605 .

[0123] The display assembly 630 displays content to the user based on the data received from the console 615. The display assembly 630 displays content using one or more display elements (e.g., display element 120). The display element can be, for example, an electronic display. In various embodiments, the display assembly 630 includes a single display element or multiple display elements (e.g., a display for each eye of the user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof. It should be noted that in some embodiments, the display element 120 can also include some or all of the functionality of the optical block 635.

[0124] The optical block 635 can amplify image light received from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or two windows of the head mounted device 605. In various embodiments, the optical block 635 includes one or more optical elements. Example optical elements included in the optical block 635 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 635 can include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 635 can have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0125] The magnification and focusing of image light by optical block 635 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content is presented using substantially all of the user's field of view (e.g., approximately 110 degrees diagonally), and in some cases, all of the user's field of view. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0126] In some embodiments, the optical block 635 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optical block 635 corrects the distortion when it receives image light generated based on the content from the electronic display.

[0127] The position sensor 640 is an electronic device that generates data indicating the position of the head mounted device 605. The position sensor 640 generates one or more measurement signals in response to the movement of the head mounted device 605. The sensor device 190 is an embodiment of the position sensor 640. Examples of the position sensor 640 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, or some combination thereof. The position sensor 640 may include multiple accelerometers that measure translational movement (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational movement (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates an estimated position of the head mounted device 605 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on the head mounted device 605. A reference point is a point that can be used to describe the position of the head mounted device 605. Although reference points may generally be defined as points in space, in practice reference points are defined as points within the head mounted device 605.

[0128] The DCA 645 generates depth information for a portion of the local area. The DCA includes a DCA controller and one or more imaging devices. The DCA 645 may also include an illuminator. The operation and structure of the DCA 645 are described above with respect to Figure 1A Described.

[0129] The audio system 650 provides audio content to the user of the head mounted device 605. The audio system 650 is an embodiment of the audio system 200 described above. The audio system 650 may include one or more acoustic sensors, one or more transducers, and an audio controller. The audio system 650 may provide spatialized audio content to the user. In some embodiments, the audio system 650 may request acoustic parameters from the mapping server 625 through the network 620. The acoustic parameters describe one or more acoustic characteristics (e.g., room impulse response, reverberation time, reverberation level, etc.) of a local area. The audio system 650 may provide information describing at least a portion of the local area from, for example, the DCA 645 and / or provide positioning information of the head mounted device 605 from the position sensor 640. The audio system 650 may generate one or more sound filters using the one or more acoustic parameters received from the mapping server 625, and use the sound filters to provide audio content to the user.

[0130] The audio system 650 may also determine one or more acoustic transfer functions for the user based on the sound transmitted within the local area of ​​the head mounted device 605. The determined acoustic transfer function may be based on the size, shape, and / or density of a portion of the user's body (e.g., the user's head, face, ears, torso, etc.), and thus the determined acoustic transfer function may be different for different wearers of the head mounted device 605. In some embodiments, the determined acoustic transfer function may be used to provide personalized audio content to the user. In addition, the acoustic transfer function may be used to identify or authenticate the user by comparing the determined function with an acoustic transfer function previously determined and associated with a known user. Based on the identification or authentication, one or more security functions (e.g., logging the user into a user account, blocking the user from logging into a user account, allowing or blocking the user from accessing personalized data, etc.) may be performed.

[0131] The I / O interface 610 is a device that allows a user to send an action request and receive a response from the console 615. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end the acquisition of image or video data, or an instruction to perform a specific action within an application. The I / O interface 610 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 615. The action request received by the I / O interface 610 is transmitted to the console 615, and the console performs the action corresponding to the action request. In some embodiments, the I / O interface 610 includes an IMU that collects calibration data indicating the estimated position of the I / O interface 610 relative to the initial position of the I / O interface 610. In some embodiments, the I / O interface 610 may provide tactile feedback to the user according to the instruction received from the console 615. For example, tactile feedback is provided when an action request is received, or when console 615 transmits instructions to I / O interface 610 that cause I / O interface 610 to generate tactile feedback when console 615 performs the action.

[0132] The console 615 provides content to the headset 605 for processing based on information received from one or more of: the DCA 645, the headset 605, and the I / O interface 610. Figure 6 In the example shown, the console 615 includes an application store 655, a tracking module 660, and an engine 665. Some embodiments of the console 615 have a Figure 6 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Figure 6The described manner is distributed among the components of the console 615. In some embodiments, the functionality discussed herein with respect to the console 615 can be implemented in the headset 605 or in a remote system.

[0133] The application storage 655 stores one or more applications for execution by the console 615. An application is a set of instructions that, when executed by the processor, generates content for presentation to the user. The content generated by the application may be responsive to input received from the user via movement of the head mounted device 605 or the I / O interface 610. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0134] The tracking module 660 tracks the movement of the head mounted device 605 or the I / O interface 610 using information from the DCA 645, one or more position sensors 640, or some combination thereof. For example, the tracking module 660 determines the location of a reference point of the head mounted device 605 in a map of the local area based on the information from the head mounted device 605. The tracking module 660 may also determine the location of an object or a virtual object. Additionally, in some embodiments, the tracking module 660 may use portions of the data from the position sensors 640 indicating the location of the head mounted device 605 and the representation of the local area from the DCA 645 to predict the future position of the head mounted device 605. The tracking module 660 provides the estimated or predicted future location of the head mounted device 605 or the I / O interface 610 to the engine 665.

[0135] The engine 665 executes the application and receives the position information, acceleration information, velocity information, predicted future position or some combination thereof of the head mounted device 605 from the tracking module 660. Based on the received information, the engine 665 determines the content provided to the head mounted device 605 for presentation to the user. For example, if the received information indicates that the user is looking to the left, the engine 665 generates content for the head mounted device 605 that reflects the user's movement in a virtual local area or in a local area that enhances the local area with additional content. In addition, the engine 665 performs an action within an application executed on the console 615 in response to an action request received from the I / O interface 610 and provides feedback to the user that the action is performed. The feedback provided may be visual or auditory feedback via the head mounted device 605, or tactile feedback via the I / O interface 610.

[0136] The network 620 couples the headset 605 and / or the console 615 to the mapping server 625. The network 620 may include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, the network 620 may include the Internet and a mobile phone network. In one embodiment, the network 620 uses standard communication technologies and / or protocols. Therefore, the network 620 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 620 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged through the network 620 may be represented using techniques and / or formats including image data in binary form (e.g., portable network graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or part of the link may be encrypted using conventional encryption techniques, such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.

[0137] The mapping server 625 may include a database storing virtual models describing a plurality of spaces, wherein a location in the virtual model corresponds to the current configuration of the local area of ​​the head mounted device 605. The mapping server 625 receives information describing at least a portion of the local area and / or location information of the local area from the head mounted device 605 via the network 620. The mapping server 625 determines a location in the virtual model associated with the local area of ​​the head mounted device 605 based on the received information and / or location information. The mapping server 625 determines (e.g., retrieves) one or more acoustic parameters associated with the local area based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The mapping server 625 may send the location of the local area and any acoustic parameter values ​​associated with the local area to the head mounted device 605.

[0138] Additional configuration information

[0139] The foregoing description of the embodiments of the present disclosure is presented for illustrative purposes; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art can recognize that many modifications and variations are possible in light of the above disclosure.

[0140] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0141] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer readable medium, the computer readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0142] Embodiments of the present disclosure may also relate to apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0143] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0144] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the inventive subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the disclosure, which is set forth in the appended claims.

Claims

1. An audio system, comprising: a sensor array comprising a plurality of acoustic sensors configured to detect sound transmitted within a local area of ​​the audio system; and A controller configured to: determining a source localization of the detected sound based on a known positional relationship between the plurality of acoustic sensors and using one or more direction of arrival (DoA) estimates generated for the sound detected by the plurality of acoustic sensors, the source localization indicating a location within the local area relative to a position of the audio system; determining at least one acoustic transfer function based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of a user of the audio system; associating the determined at least one acoustic transfer function with the determined source localization; identifying at least one stored acoustic transfer function corresponding to the authorized user, the at least one stored acoustic transfer function selected from a plurality of stored acoustic transfer functions corresponding to the authorized user based on the determined source location; and The user is identified based on a comparison of the determined at least one acoustic transfer function and the identified at least one stored acoustic transfer function.

2. The audio system of claim 1, wherein: The plurality of acoustic sensors include a first acoustic sensor configured to generate a first audio signal based on the sound and a second acoustic sensor configured to generate a second audio signal based on the sound; and The controller is configured to determine the at least one acoustic transfer function by determining a ratio between the first audio signal and the second audio signal.

3. The audio system according to claim 2, wherein: The controller is configured to determine a set of acoustic transfer functions, each acoustic transfer function being based on a ratio between two acoustic sensors of the plurality of acoustic sensors, and the controller is configured to identify the user based on the determined set of acoustic transfer functions.

4. The audio system according to claim 1, wherein: The plurality of acoustic sensors includes a pair of acoustic sensors located proximate each ear of the user, and the acoustic transfer function corresponds to a head-related transfer function (HRTF).

5. The audio system of claim 1, wherein: The sensor array is located on a head mounted device.

6. The audio system of claim 5, further comprising a proximity sensor, and wherein the controller is further configured to: determining whether the user is wearing the head mounted device based on input received from the proximity sensor; and In response to determining that the user is wearing the headgear, the at least one acoustic transfer function is determined.

7. The audio system of claim 1, wherein: The identified at least one stored acoustic transfer function is associated with a source location that is within a threshold distance of the determined source location.

8. The audio system according to claim 1, in, The controller is further configured to cause the speaker to emit the sound.

9. The audio system of claim 1, wherein: The sound is generated by the user.

10. The audio system of claim 1, wherein: The controller is also configured to: One or more security functions are performed based on a result of the comparison, wherein the one or more security functions include authorizing the user to access personalized data or setting a configuration of the wearable device based on one or more stored user preferences.

11. The audio system of claim 1, wherein: The controller is also configured to: extracting a portion of the detected sound corresponding to the sound received from the source location within a specified time window starting from receiving a first portion of the direct sound of the detected sound by the plurality of acoustic sensors; The at least one acoustic transfer function is determined based on the extracted portion of the detected sound.

12. The audio system of claim 1, wherein: The source location is determined based on triangulation of the DoA estimate.

13. The audio system of claim 1, wherein: Identifying at least one stored acoustic transfer function includes identifying a stored acoustic transfer function of the plurality of stored acoustic transfer functions associated with a source location within at least a threshold distance of the determined source location.

14. The audio system of claim 1, wherein: The controller is further configured to prompt the user to move within the local area to detect sounds associated with a plurality of different source locations.

15. A method for an audio system, comprising: detecting sound transmitted within a localized area of ​​the headset using a sensor array, the sensor array comprising a plurality of acoustic sensors; determining a source localization of the detected sound based on a known positional relationship between the plurality of acoustic sensors and using one or more direction of arrival (DoA) estimates generated for the sound detected by the plurality of acoustic sensors, the source localization indicating a location within the local area relative to a position of the audio system; determining at least one acoustic transfer function associated with the user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of the user of the head mounted device; associating the determined at least one acoustic transfer function with the determined source localization; identifying at least one stored acoustic transfer function corresponding to the authorized user, the at least one stored acoustic transfer function selected from a plurality of stored acoustic transfer functions corresponding to the authorized user based on the determined source location; and The user is identified based on a comparison of the determined at least one acoustic transfer function and the identified at least one stored acoustic transfer function.

16. The method according to claim 15, in: The plurality of acoustic sensors include a first acoustic sensor configured to generate a first audio signal based on the sound and a second acoustic sensor configured to generate a second audio signal based on the sound; and Wherein determining the at least one acoustic transfer function comprises determining a ratio between the first audio signal and the second audio signal.

17. The method according to claim 15, further comprising: determining whether the user is wearing the head mounted device based on input received from a proximity sensor of the head mounted device; and In response to determining that the user is wearing the headgear, the at least one acoustic transfer function is determined.

18. The method according to claim 15, wherein: The at least one stored acoustic transfer function is associated with a source location that is within a threshold distance of the determined source location.

19. The method according to claim 15, further comprising: One or more security functions are performed based on a result of the comparison, wherein the one or more security functions include authorizing the user to access personalized data or setting a configuration of the head mounted device based on one or more stored user preferences.

20. A non-transitory computer readable medium storing instructions that, when executed by one or more processors of an audio system including a sensor array located on a head mounted device, cause the one or more processors to perform operations comprising: detecting sound transmitted within a localized area of ​​the headset using a sensor array, the sensor array comprising a plurality of acoustic sensors; determining a source localization of the detected sound based on a known positional relationship between the plurality of acoustic sensors and using one or more direction of arrival (DoA) estimates generated for the sound detected by the plurality of acoustic sensors, the source localization indicating a location within the local area relative to a position of the audio system; determining at least one acoustic transfer function associated with the user based in part on the detected sound, wherein the at least one acoustic transfer function defines a transformation of the sound caused in part by a head of the user of the head mounted device; associating the determined at least one acoustic transfer function with the determined source localization; identifying at least one stored acoustic transfer function corresponding to the authorized user, the at least one stored acoustic transfer function selected from a plurality of stored acoustic transfer functions corresponding to the authorized user based on the determined source location; and The user is identified based on a comparison of the determined at least one acoustic transfer function and the identified at least one stored acoustic transfer function.

Images