Audio profiles for personalized audio enhancement

Abstract

The system creates an audio profile. The audio profile can be stored in a database. For example, the audio profile can be securely stored in a database of a social network and associated with a user account. The audio profile can contain data describing the way a particular user hears and interprets sound. Systems and applications that present sound to the user can access the audio profile and modify the sound presented to the user based on the data in the audio profile to enhance the user's audio experience.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 62 / 857,410, filed on June 5, 2019, and U.S. Application No. 16 / 562,879, filed on September 6, 2019. The contents of U.S. Application No. 62 / 857,410 and U.S. Application No. 16 / 562,879 are incorporated herein by reference in their entirety for all purposes. Field of the Invention

[0003] The present disclosure relates generally to audio systems, and particularly to providing customized audio content.

[0004] background

[0005] People hear sounds in different ways. For users of an audio system, such as an audio system in an artificial reality system, the sounds presented by the audio system may be heard in different ways by different users. For example, some users may suffer from hyperacusis (oversensitivity), presbycusis (insensitivity to certain frequencies), or some other condition that affects the way they perceive sounds. Therefore, auditory differences between different users may affect their perception of the presented audio content.

[0006] Overview

[0007] The wearable device performs an audio assessment for the user. Based on the results of the audio assessment, the wearable device creates an audio profile. The audio profile can be stored in a database. For example, the audio profile can be stored in a database of a social network and associated with a user account. The audio profile can contain data describing the way a specific user hears and interprets sound. For example, many users have difficulty understanding speech in a highly reverberant environment. Systems and applications that present sound to the user can access the audio profile and modify the sound presented to the user based on the data in the audio profile to enhance the user's audio experience.

[0008] In some embodiments, systems, methods, and computer program products may perform operations including collecting audio evaluation data for a user by an audio system on a wearable device. The audio evaluation data may be collected using active and / or passive audio evaluation routines. The audio system creates an audio profile based on the audio evaluation data and physical data describing the user. The audio system may store the audio profile locally on the wearable device or transmit the audio file to an external system, such as a social network. The audio system presents audio content to the user based in part on the audio profile.

[0009] According to the present invention, a method is provided, the method comprising collecting, by a wearable device, audio evaluation data of a user; updating an audio profile based on the audio evaluation data and physical data describing the user; and presenting audio content to the user based in part on the audio profile.

[0010] Optionally, the method further comprises transmitting the audio profile to a social network, wherein the social network associates the audio profile with the user profile. Optionally, the social network associates the audio profile with an enhancement scheme.

[0011] Optionally, the method further comprises creating an audio profile by the wearable device. Optionally, the method further comprises receiving audio content from an external system and modifying the audio content based on the audio profile.

[0012] Optionally, collecting audio evaluation data comprises presenting a sound to a user, and detecting a user's response to the sound. Optionally, collecting audio evaluation data comprises determining a location of a sound source in a local area, and detecting a user's response to the sound source. Optionally, collecting audio evaluation data comprises performing an audio evaluation routine to determine parameters that characterize the user's hearing. Optionally, the parameters comprise at least one of a scaling factor or a maximum reverberation level.

[0013] Optionally, the body data describes the shape of the user's ears.

[0014] Optionally, the method further comprises retrieving an audio profile from a social network.

[0015] Optionally, the audio profile includes security settings, wherein the security settings indicate whether a third party is authorized to access the audio profile.

[0016] According to the present invention, a computer program product is also provided, which includes a non-transitory computer-readable storage medium containing computer program code, which is used to collect audio evaluation data of a user through a wearable device, update an audio profile based on the audio evaluation data and physical data describing the user, and present audio content to the user based in part on the audio profile.

[0017] Optionally, the computer program product further comprises computer program code for transmitting the audio profile to a social network, wherein the social network associates the audio profile with a user profile.

[0018] According to the present invention, a wearable device is also provided, which includes a non-transitory computer-readable storage medium containing computer program code, which is used for collecting audio evaluation data of a user by the wearable device, updating an audio profile based on the audio evaluation data and physical data describing the user, and presenting audio content to the user based in part on the audio profile.

[0019] Optionally, the wearable device further comprises computer program code for transmitting the audio profile to a social network, wherein the social network associates the audio profile with the user profile. Optionally, the social network associates the audio profile with an enhancement scheme.

[0020] Optionally, the wearable device further comprises computer program code for receiving audio content from an external system and modifying the audio content based on the audio profile. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A is a perspective view of a headset implemented as an eyewear device according to one or more embodiments.

[0023] Figure 1B is a perspective view of a head mounted device implemented as a head mounted display according to one or more embodiments.

[0024] Figure 2 is a block diagram of an audio system according to one or more embodiments.

[0025] Figure 3 A flow diagram illustrating a process for providing personalized audio enhancement in accordance with one or more embodiments.

[0026] Figure 4 A system including a head mounted device according to one or more embodiments.

[0027] The accompanying drawings depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0028] Detailed Description

[0029] A system provides personalized audio content to a user to enhance the user's audio experience. Wearable devices such as head-mounted devices or smart watches can perform audio assessments for users. For example, wearable devices can present audio games, audiology screening (audiology screen), hearing tests, instantaneous environmental assessments, and post-experience surveys to users. Wearable devices can monitor the user's behavior in response to audio content, such as eye gaze direction in response to sound, whether the user reacts to the sound, whether the user understands the voice in the audio content, etc. The system can create an audio profile. The audio profile can be stored in a database. For example, the audio profile can be stored in a database of a social network and associated with a user account. The audio profile can contain data describing the way a specific user hears and interprets the sound. The system and application that presents sound to the user can access the audio profile and modify the sound presented to the user based on the data in the audio profile to enhance the user's audio experience. For example, if the user is hypersensitive to the frequency between 10kHz-15kHz, the sound amplitude in this frequency band can be reduced for the user.

[0030] Traditional audiological screening can understand pure tone hearing threshold (audibility threshold), but this assessment is carried out in absolute silence, and may not necessarily provide insight into how the listener will perceive or perform in a real environment. The system and method described herein monitor and evaluate the hearing performance of the user in a real-life scenario. In addition, these routines can be combined with other results related to the user's physical characteristics, such as personalized head-related transfer functions and body-related transfer functions, to generate a unique personal audio profile. Once the audio profile is established, it can be attached and maintained as part of the user's social network profile. The audio profile can be encrypted or otherwise protected to protect the user's medical data. The system can prompt the user to allow the audio profile to be stored, and the system can allow the user to select who can access the audio profile. The audio profile can be accessed to improve the audio of all audio experiences of the user, including video calls, virtual reality games and applications, augmented reality telepresence (telepresence), real-time voice enhancement, real-time noise reduction, etc.

[0031] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used to create content in artificial reality and / or be used in other ways in artificial reality. Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted devices) connected to a host computer system, standalone wearable devices (e.g., head-mounted devices), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0032] Figure 1A is a perspective view of a head mounted device 100 implemented as an eyewear device according to one or more embodiments. In some embodiments, the eyewear device is a near eye display (NED). Typically, the head mounted device 100 can be worn on a user's face such that content (e.g., media content) is presented using a display component and / or an audio system. However, the head mounted device 100 can also be used such that media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 100 include one or more images, videos, audio, or some combination thereof. The head mounted device 100 includes a frame and may include, among other components, a display component including one or more display elements 120, a depth camera assembly (DCA), an audio system, and a position sensor 190. Although Figure 1A The example positioning on the head mounted device 100 shows components of the head mounted device 100, but these components may be located elsewhere on the head mounted device 100, on a peripheral device that is paired with the head mounted device 100, or some combination of these two locations. Similarly, components on the head mounted device 100 may be located on a smaller device than on the head mounted device 100. Figure 1A More or less than shown.

[0033] The frame 110 holds the other components of the head mounted device 100. The frame 110 includes a front portion that holds one or more display elements 120 and an end piece (e.g., temple) that is attached to the user's head. The front portion of the frame 110 bridges the top of the user's nose. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that curls behind the user's ear (e.g., temple tip, ear hook).

[0034] One or more display elements 120 provide light to a user wearing the head mounted device 100. As shown, the head mounted device includes a display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light provided to the eyebox of the head mounted device 100. The eyebox is the spatial location occupied by the eyes when the user wears the head mounted device 100. For example, the display element 120 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional light source, one or more linear light sources, one or more point light sources, etc.) and one or more waveguides. The light from the light source is coupled inwardly into one or more waveguides, which output light in such a way that there is a pupil replication in the eyebox of the head mounted device 100. The inward coupling and / or outward coupling of light from one or more waveguides can be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a reflector, etc.) that scans the light from the light source when the light from the light source is coupled inwardly into one or more waveguides. Note that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a local area around the head mounted device 100. The local area is the area around the head mounted device 100. For example, the local area may be a room where a user wearing the head mounted device 100 is located, or the user wearing the head mounted device 100 may be outside and the local area is an external area. In this case, the head mounted device 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent so that light from the local area can be combined with light from one or more display elements to produce AR and / or MR content.

[0035] In some embodiments, display elements 120 do not generate image light, but rather lenses transmit light from localized areas to the viewport. For example, one or both of display elements 120 may be non-corrective lenses (over-the-counter), or prescription lenses (e.g., single vision lenses, bifocal and trifocal lenses, or progressive lenses) to help correct for imperfections in the user's vision. In some embodiments, display elements 120 may be polarized and / or tinted to protect the user's eyes from sunlight.

[0036] It should be noted that in some embodiments, the display element 120 may include an additional optical block (not shown). The optical block may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that direct light from the display element 120 to the viewing window. The optical block may, for example, correct aberrations in some or all image content, magnify some or all images, or some combination thereof.

[0037] The display element 120 may display content as part of the audio assessment routine. For example, the display element 120 may display instructions or provide feedback to the user during the audio assessment routine.

[0038] The DCA determines depth information for a portion of a local area around the head mounted device 100. The DCA includes one or more imaging devices 130 and a DCA controller (not shown). Figure 1A ), and may also include an illuminator 140. In some embodiments, the illuminator 140 illuminates a portion of the local area with light. The light may be, for example, structured light (e.g., dot patterns, stripes, etc.) in infrared (IR), IR flash for time of flight, etc. In some embodiments, one or more imaging devices 130 capture an image of a portion of the local area including light from the illuminator 140. As shown in the figure, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, there is no illuminator 140 and at least two imaging devices 130.

[0039] The DCA controller calculates depth information for a portion of the local area using the captured images and one or more depth determination techniques. The depth determination techniques may be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from illuminator 140), some other technique for determining the depth of a scene, or some combination thereof.

[0040] The DCA controller can determine the location of one or more sound sources in the local scene in conjunction with the audio system. The DCA controller can provide the location of the object in the local scene. The DCA can provide the location of the object to the audio system. The audio system can determine that sound sources emanating from the same direction of the object located by the DCA may be caused by the object. The audio system can then use the determined sound source location in conjunction with the audio evaluation routine.

[0041] The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller 150. However, in other embodiments, the audio system may include different and / or additional components. Similarly, in some cases, the functions described with reference to the components of the audio system may be distributed between the components in a manner different from that described herein. For example, some or all functions of the controller may be performed by a remote server.

[0042] The audio system can perform an audio assessment routine for the user. The audio assessment routine is a series of steps by which the audio system determines parameters that characterize the user's hearing, such as hyperacusis or presbycusis to certain frequencies, acceptable reverberation levels, etc. The audio assessment routine can include monitoring user responses using various sensors on the head mounted device 100 (e.g., imaging device 130, acoustic sensor 180, or position sensor 190). The audio system creates an audio profile for the user based on the parameters. An audio profile is a set of parameters that describe how a user hears sounds. The audio system can use the audio profile to modify the sounds presented to the user.

[0043] The transducer array presents sound to the user. The transducer array includes multiple transducers. The transducer can be a speaker 160 or a tissue transducer 170 (e.g., a bone conduction transducer or a cartilage conduction transducer). Although the speaker 160 is shown outside the frame 110, the speaker 160 can be enclosed in the frame 110. In some embodiments, instead of a separate speaker for each ear, the head mounted device 100 includes a speaker array that includes multiple speakers integrated into the frame 110 to improve the directionality of the presented audio content. The tissue transducer 170 is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to generate sound. The number and / or positioning of the transducers can be different from the head mounted device 100. Figure 1A Different than shown.

[0044] The sensor array detects sounds in a local area of ​​the head mounted device 100. The sensor array includes a plurality of acoustic sensors 180. The acoustic sensors 180 capture sounds emitted from one or more sound sources in a local area (e.g., a room). Each acoustic sensor is configured to detect sounds and convert the detected sounds into an electronic format (analog or digital). The acoustic sensor 180 may be a sound wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sounds.

[0045] In some embodiments, one or more acoustic sensors 180 can be placed in the ear canal of each ear (e.g., acting as binaural microphones). In some embodiments, acoustic sensors 180 can be placed on an exterior surface of head mounted device 100, on an interior surface of head mounted device 100, separate from head mounted device 100 (e.g., as part of some other device), or some combination thereof. The number and / or positioning of acoustic sensors 180 can vary depending on the device. Figure 1A For example, the number of acoustic detection positionings can be increased to increase the amount of audio information collected and the sensitivity and / or accuracy of the information. The acoustic detection positionings can be oriented so that the microphones can detect sounds in a wide range of directions around the user wearing the head mounted device 100.

[0046] The audio controller 150 processes information from the sensor array describing the sounds detected by the sensor array. The audio controller 150 may include a processor and a computer-readable storage medium. The audio controller 150 may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and / or a head-related transfer function), track the location of a sound source, form a beam in the direction of a sound source, classify a sound source, generate a sound filter for the speaker 160, or some combination thereof.

[0047] The position sensor 190 generates one or more measurement signals in response to the movement of the head mounted device 100. The position sensor 190 may be located on a portion of the frame 110 of the head mounted device 100. The position sensor 190 may include an inertial measurement unit (IMU). Examples of the position sensor 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor 190 may be located outside the IMU, inside the IMU, or some combination thereof.

[0048] In some embodiments, the head mounted device 100 can provide simultaneous localization and mapping (SLAM) of the position of the head mounted device 100 and updates of local area models. For example, the head mounted device 100 can include a passive camera assembly (PCA) that generates color image data. The PCA can include one or more RGB cameras that are used to capture images of some or all local areas. In some embodiments, some or all of the imaging devices 130 of the DCA can also be used as PCAs. The images captured by the PCA and the depth information determined by the DCA can be used to determine parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the position sensor 190 tracks the position (e.g., positioning and posture) of the head mounted device 100 in the room. The following is combined with Figure 4 Additional details regarding components of head mounted device 100 are discussed.

[0049] The head mounted device 100 includes an eye tracking unit 195. The eye tracking unit 195 may include one or more cameras that capture images of the user's eyes. The eye tracking unit 195 may also include one or more illuminators that illuminate the user's eyes. The eye tracking unit 195 estimates the angular orientation of the user's eye or eyes. In some embodiments, the eye tracking unit 195 may detect distortion in the illumination pattern projected by the illuminator to determine the angular orientation of the user's eyes. The orientation of the eyes corresponds to the direction in which the user is staring within the head mounted device 100. The orientation of the user's eyes may be in the direction of the foveal axis, which is the axis between the fovea (the area on the retina of the eye where the most photoreceptor cells are concentrated) and the center of the pupil of the eye. Typically, when the user's eyes are fixed on a point, the foveal axis of the user's eyes intersects the point. The pupil axis is another axis of the eye, which is defined as an axis perpendicular to the corneal surface that passes through the center of the pupil. Generally, the pupil axis is not directly aligned with the foveal axis. The two axes intersect at the center of the pupil, but the orientation of the foveal axis is offset from the pupil axis by approximately -1° to 8° laterally and approximately ±4° vertically. Because the foveal axis is defined based on the fovea, which is located at the back of the eye, in some eye tracking embodiments, the foveal axis may be difficult or impossible to detect directly. Therefore, in some embodiments, the orientation of the pupil axis is detected, and the foveal axis is estimated based on the detected pupil axis. However, in some embodiments, the orientation of the pupil axis can be used to estimate the angular orientation of the user's eye or eyes without adjusting for foveal axis differences.

[0050] Typically, the movement of the eye corresponds not only to the angular rotation of the eye, but also to the translation of the eye, the change in the torsion of the eye, and / or the change in the shape of the eye. The eye tracking unit 195 can also detect the translation of the eye: that is, the change in the position of the eye relative to the eye socket. In some embodiments, the translation of the eye is not directly detected, but is approximated based on a mapping from the detected angular orientation. The eye translation corresponding to the change in the position of the eye relative to the detection component of the eye tracking unit can also be detected. This type of translation may occur, for example, due to the movement of the position of the head-mounted device 100 on the user's head. The eye tracking unit 195 can also detect the torsion of the eye, that is, the rotation of the eye around the pupil axis. The eye tracking unit 195 can use the detected eye torsion to estimate the orientation of the foveal axis relative to the pupil axis. The eye tracking unit 195 can also track changes in the shape of the eye, which can be approximated as a skew or scale linear transformation or a distortion deformation (for example, due to torsion deformation). The eye tracking unit 195 can estimate the foveal axis based on some combination of the angular orientation of the pupil axis, the translation of the eye, the torsion of the eye, and the current shape of the eye.

[0051] In some embodiments, the eye tracking unit 195 may include at least one emitter that projects a structured light pattern on all or part of the eye. The pattern is then projected onto the shape of the eye, which may produce perceptible distortion in the structured light pattern when viewed from an offset angle. The eye tracking unit 195 may also include at least one camera that detects distortion, if any, of the light pattern projected onto the eye. A camera oriented on an axis different from the emitter captures the illumination pattern on the eye. This process is referred to herein as "scanning" the eye. By detecting deformations of the illumination pattern on the surface of the eye, the eye tracking unit 195 can determine the shape of the scanned portion of the eye. Thus, the captured distorted light pattern indicates the 3D shape of the illuminated portion of the eye. By deriving the 3D shape of the portion of the eye illuminated by the emitter, the orientation of the eye can be derived. The eye tracking unit can also estimate the pupil axis, the translation of the eye, the torsion of the eye, and the current shape of the eye based on the image of the illumination pattern captured by the camera.

[0052] In other embodiments, any suitable type of eye tracking system may be used. For example, the eye tracking unit 195 may capture images of the eye, capture stereo images of the eye, may utilize a ring of LEDs around the eye that illuminate in sequence and determine eye orientation based on reflections from the LEDs, may utilize time of flight measurements, etc.

[0053] Because the orientation can be determined for both eyes of the user, the eye tracking unit 195 is able to determine where the user is looking. The head mounted device 100 can use the orientation of the eyes to, for example, determine the user's interpupillary distance (IPD), determine the gaze direction, introduce depth cues (e.g., blur images outside the user's primary line of sight), collect heuristics about user interactions in VR media (e.g., time spent on any particular subject, object, or frame based on exposure to stimuli), some other functionality based in part on the orientation of at least one of the user's eyes, or some combination thereof. Determining the direction of the user's gaze may include determining a point of convergence based on the determined orientation of the user's left and right eyes. The point of convergence may be the point where the two foveal axes of the user's eyes intersect (or the closest point between the two axes). The direction of the user's gaze may be the direction of a line passing through the point of convergence and through the midpoint of the pupils of the user's eyes. The following is combined with Figure 4 Additional details regarding components of head mounted device 100 are discussed.

[0054] The audio system generates an audio profile for the user. The audio system monitors the user's response to the sound. The eye tracking unit 195 detects the gaze localization of the user's eyes in response to the sound. The audio system measures the accuracy of various parameters of the sound based on the user's response, such as the difference between gaze localization and sound source localization. The audio system uses the audio profile to subsequently present enhanced audio content to the user. Reference Figure 2-Figure 4 The audio enhancement process is further described.

[0055] Figure 1B 1 is a perspective view of a head mounted device 105 implemented as an HMD according to one or more embodiments. In embodiments describing an AR system and / or an MR system, a portion of the front side of the HMD is at least partially transparent in the visible light band (approximately 380 nm to 750 nm), and a portion of the HMD between the front side of the HMD and the user's eyes is at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a strap 175. The head mounted device 105 includes many of the same components as described above with reference to FIG. Figure 1A The same components described above are modified to integrate with the HMD form factor. For example, the HMD includes a display assembly, a DCA, an audio system, and a position sensor 190. Figure 1B Shown are an illuminator 140, a plurality of speakers 160, a plurality of imaging devices 130, a plurality of acoustic sensors 180, and a position sensor 190. The speakers 160 may be located in various locations, such as coupled to the band 175 (as shown), coupled to the front rigid body 115, or may be configured to be inserted into the ear canal of a user.

[0056] Figure 2is a block diagram of an audio system 200 according to one or more embodiments. Figure 1A or Figure 1B The audio system in may be an embodiment of the audio system 200. The audio system 200 generates a customized audio profile for the user. The audio system 200 may then use the audio profile to generate audio content for the user. Figure 2 In the embodiment of the present invention, the audio system 200 includes a transducer array 210, a sensor array 220, and an audio controller 230. Some embodiments of the audio system 200 have different components than those described here. Similarly, in some cases, functions can be distributed between components in a manner different from that described here.

[0057] The transducer array 210 is configured to present audio content. The transducer array 210 includes a plurality of transducers. A transducer is a device that provides audio content. The transducer may be, for example, a speaker (e.g., speaker 160), a tissue transducer (e.g., tissue transducer 170), some other device that provides audio content, or some combination thereof. The tissue transducer may be configured to be used as a bone conduction transducer or a cartilage conduction transducer. The transducer array 210 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, the transducer array 210 may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range, while a moving coil transducer may be used to cover a second part of a frequency range.

[0058] The bone conduction transducer generates sound pressure waves by vibrating the bones / tissue of the user's head. The bone conduction transducer can be coupled to a portion of the head mounted device and can be configured to couple to a portion of the user's skull behind the auricle. The bone conduction transducer receives vibration instructions from the audio controller 230 and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate tissue-propagated sound pressure waves that bypass the eardrum and propagate toward the user's cochlea.

[0059] The bone conduction transducer generates sound pressure waves by vibrating one or more portions of the ear cartilage of the user's ear. The bone conduction transducer can be coupled to a portion of the head-mounted device and can be configured to couple to one or more portions of the ear cartilage of the ear. For example, the bone conduction transducer can be coupled to the back of the auricle of the user's ear. The bone conduction transducer can be located anywhere along the ear cartilage around the outer ear (e.g., the auricle, the tragus, some other portion of the ear cartilage, or some combination thereof). Vibrating one or more portions of the ear cartilage can generate: air-borne sound pressure waves outside the ear canal; sound pressure waves generated by tissue causing certain portions of the ear canal to vibrate, thereby generating air-borne sound pressure waves within the ear canal; or some combination thereof. The generated air-borne sound pressure waves propagate along the ear canal toward the eardrum.

[0060] The transducer array 210 generates audio content according to instructions from the audio controller 230. The audio content can be generated in conjunction with an audio evaluation routine. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that sounds like it originates from a specific direction and / or target area (e.g., objects and / or virtual objects in a local area). For example, the spatialized audio content can make the sound sound like a virtual singer from across the room from the user of the audio system 200. The transducer array 210 can be coupled to a wearable device (e.g., head mounted device 100 or head mounted device 105). In an alternative embodiment, the transducer array 210 can be a plurality of speakers that are separate from the wearable device (e.g., coupled to an external console).

[0061] The sensor array 220 detects sounds in a local area around the sensor array 220. The sensor array 220 may include a plurality of acoustic sensors, each of which detects air pressure changes of sound waves and converts the detected sound into an electronic format (analog or digital). The plurality of acoustic sensors may be located on a head-mounted device (e.g., head-mounted device 100 and / or head-mounted device 105), a user (e.g., in the ear canal of the user), a neckband, or some combination thereof. The acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, the sensor array 220 is configured to monitor the audio content generated by the transducer array 210 using at least some of the plurality of acoustic sensors. Increasing the number of sensors may improve the accuracy of information (e.g., directionality) describing the sound field generated by the transducer array 210 and / or the sound from the local area.

[0062] The audio controller 230 controls the operation of the audio system 200. Figure 2In an embodiment of the present invention, the audio controller 230 includes a data storage 235, a DOA estimation module 240, a transfer function module 250, a tracking module 260, a beamforming module 270, a sound filter module 280, and a personalization module 290. The audio controller 230 can be located inside a wearable device such as a head-mounted device. Some embodiments of the audio controller 230 have different components than described here. Similarly, the functions can be distributed among the components in a manner different from that described here. For example, some functions of the controller can be performed outside the head-mounted device.

[0063] The data storage 235 stores data for use by the audio system 200. The data in the data storage 235 may include sounds recorded in a local area of ​​the audio system 200, audio content, head-related transfer functions (HRTFs), transfer functions of one or more sensors, array transfer functions (ATFs) of one or more acoustic sensors, sound source localization, virtual models of the local area, direction of arrival estimates, sound filters, and other data related to the use of the audio system 200, or any combination thereof. The data storage 235 may include audio profiles of one or more users, instructions for the audio evaluation routine, user response data for the audio evaluation routine, and the like.

[0064] The DOA estimation module 240 is configured to locate sound sources in a local area based in part on information from the sensor array 220. Localization is the process of determining the location of a sound source relative to the user of the audio system 200. The DOA estimation module 240 performs a DOA analysis to locate one or more sound sources in a local area. The DOA analysis may include analyzing the intensity, spectrum, and / or arrival time of each sound at the sensor array 220 to determine the direction from which the sound originated. In some cases, the DOA analysis may include any suitable algorithm for analyzing the ambient acoustic environment in which the audio system 200 is located.

[0065] For example, the DOA analysis may be designed to receive an input signal from the sensor array 220 and apply a digital signal processing algorithm to the input signal to estimate the direction of arrival. These algorithms may include, for example, a delay and sum algorithm in which the input signal is sampled and the final weighted and delayed versions of the sampled signals are averaged together to determine the DOA. A least mean square (LMS) algorithm may also be implemented to create an adaptive filter. The adaptive filter may then be used, for example, to identify differences in signal strength or differences in arrival time. These differences may then be used to estimate the DOA. In another embodiment, the DOA may be determined by converting the input signal into the frequency domain and selecting a specific bin in the time-frequency (TF) domain to be processed. Each selected TF bin may be processed to determine whether the bin includes a portion of the audio spectrum having a direct path audio signal. Those bins having a portion of the direct path signal may then be analyzed to identify the angle at which the sensor array 220 receives the direct path audio signal. The determined angle may then be used to identify the DOA of the received input signal. Other algorithms not listed above may also be used alone or in combination with the above algorithms to determine the DOA.

[0066] In some embodiments, the DOA estimation module 240 can also determine the DOA relative to the absolute position of the audio system 200 within the local area. The position of the sensor array 220 can be received from an external system (e.g., some other component of the head mounted device, an artificial reality console, a mapping server, a position sensor (e.g., the position sensor 190), etc.). The external system can create a virtual model of the local area, where the local area and the position of the audio system 200 are mapped. The received position information may include the location and / or orientation of some or all of the audio system 200 (e.g., the sensor array 220). The DOA estimation module 240 can update the estimated DOA based on the received position information.

[0067] The transfer function module 250 is configured to generate one or more acoustic transfer functions. In general, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function module 250 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how a microphone receives sound from a point in space.

[0068] The ATF includes multiple transfer functions that characterize the relationship between a sound source and the corresponding sound received by the acoustic sensors in the sensor array 220. Therefore, for a sound source, each acoustic sensor in the sensor array 220 has a corresponding transfer function. This set of transfer functions is collectively referred to as the ATF. Therefore, for each sound source, there is a corresponding ATF. Note that the sound source can be, for example, someone or something that produces sound in a local area, a user, or one or more transducers of the transducer array 210. The ATF positioned relative to a specific sound source of the sensor array 220 may vary from user to user, because the human anatomical structure (e.g., ear shape, shoulders, etc.) affects the sound when the sound reaches the human ear. Therefore, the ATF of the sensor array 220 is personalized for each user of the audio system 200.

[0069] In some embodiments, the transfer function module 250 determines one or more HRTFs for a user of the audio system 200. HRTFs characterize how an ear receives sound from a point in space. HRTFs positioned relative to a particular source of a person are unique to each ear of the person (and unique to a person) because the person's anatomy (e.g., ear shape, shoulders, etc.) affects the sound as it travels to the person's ears. In some embodiments, the transfer function module 250 can determine the HRTF for a user using a calibration process.

[0070] In some embodiments, transfer function module 250 may provide the ATF and HRTF to personalization module 290 for combination with audio evaluation data in the user's audio profile.

[0071] The tracking module 260 is configured to track the positioning of one or more sound sources. The tracking module 260 can compare current DOA estimates and compare them with a stored history of previous DOA estimates. In some embodiments, the audio system 200 can recalculate the DOA estimate periodically, for example once per second, or once per millisecond. The tracking module can compare the current DOA estimate with the previous DOA estimate, and in response to a change in the DOA estimate of the sound source, the tracking module 260 can determine that the sound source has moved. In some embodiments, the tracking module 260 can detect changes in positioning based on visual information received from a head-mounted device or some other external source. The tracking module 260 can track the movement of one or more sound sources over time. The tracking module 260 can store values ​​about the number of sound sources and the positioning of each sound source at each point in time. In response to changes in the number of sound sources or the values ​​of the positioning, the tracking module 260 can determine that the sound source has moved. The tracking module 260 can calculate an estimate of the positioning variance. The positioning variance can be used as a confidence level for each determination of a motion change.

[0072] The beamforming module 270 is configured to process one or more ATFs to selectively emphasize the sound from a sound source within a certain area while weakening the sound from other areas. When analyzing the sound detected by the sensor array 220, the beamforming module 270 can combine information from different acoustic sensors to emphasize the relevant sound from a specific area of ​​the local area while weakening the sound from outside the area. The beamforming module 270 can isolate the audio signal associated with the sound from a specific sound source from other sound sources in the local area based on different DOA estimates, such as from the DOA estimation module 240 and the tracking module 260. The beamforming module 270 can therefore selectively analyze discrete sound sources in the local area. In some embodiments, the beamforming module 270 can enhance the signal from the sound source. For example, the beamforming module 270 can apply a sound filter that eliminates the signal above, below, or between specific frequencies. Signal enhancement is used to enhance the sound associated with a given identified sound source relative to other sounds detected by the sensor array 220. In some embodiments, based on parameters received from personalization module 290, beamforming module 270 may adjust the width or behavior of the beam in a noisy environment according to the user's estimated speech understanding or listening effort.

[0073] The sound filter module 280 determines the sound filter for the transducer array 210. In some embodiments, the sound filter causes the audio content to be spatialized so that the audio content sounds like it originates from the target area. The sound filter module 280 can generate the sound filter using HRTF and / or acoustic parameters. The acoustic parameters describe the acoustic properties of the local area. The acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the sound filter module 280 calculates one or more acoustic parameters. In some embodiments, the sound filter module 280 requests the acoustic parameters from the mapping server (e.g., as described below with reference to Figure 4 described above).

[0074] The sound filter module 280 provides a sound filter to the transducer array 210. In some embodiments, the sound filter can cause positive or negative amplification of the sound depending on the frequency. For example, based on the parameters received from the personalization module 290, the sound filter module 280 can amplify the sound of a specific frequency that the user does not detect as easily as other frequencies.

[0075] The personalization module 290 generates an audio profile for the user. The audio profile describes how the user perceives sound. The audio profile contains parameters that describe how the sound should be presented to the user to enhance the user's audio experience. The parameters may include gain and limits for the sound. For example, these parameters may indicate a 6dB gain for the frequency band between 10kHz-15kHz, up to a maximum limit of 85dB sound pressure level, so that the audio system increases the amplitude of the sound in this frequency band by 6dB.

[0076] In some embodiments, the parameters may include a scaling factor indicating that the amplitude of certain frequencies should be increased, or that the amplitude of certain frequencies should be reduced relative to other frequencies (e.g., a scaling factor of 1.2 for a frequency band between 10kHz-15kHz may indicate that the audio system should increase the amplitude of the sound in that frequency band by 20%). These parameters may indicate the maximum reverberation level of an environment that allows the user to understand speech. These parameters may describe beamformer adjustments in a noisy environment based on the user's estimated speech understanding or listening effort, such as width or dynamic behavior. These parameters may provide artificial amplification of sound source separation to help with spatial release of masking. These parameters may provide gain and compression structures that are appropriate for the user's hearing. These parameters may describe how to enhance the voices of other speakers, such as by changing the frequency, which allows the user to better understand the speaker. Sound parameters may include sound localization capabilities, the magnitude of spatial release of masking, binaural masking level differences, speech in noise thresholds for different types of background noise, or any other suitable parameters that may increase audio quality for the user.

[0077] The audio system 200 modifies the sound presented to the user according to the parameters in the audio file to provide the user with an enhanced audio experience.For example, the parameters in the audio profile may indicate that the audio system should increase or decrease the amplitude of certain frequencies presented to the user.

[0078] To generate an audio profile for the user, the personalization module 290 performs active and / or passive audio assessments for the user. For example, the personalization module 290 may present audio games, audiology screenings, hearing tests, momentary environmental assessments, and post-experience surveys to the user.

[0079] The audio evaluation may include an audio evaluation routine. The audio evaluation routine is a series of steps by which the audio system determines parameters that characterize the user's hearing, such as hyperacusis or presbycusis to certain frequencies, acceptable reverberation levels, and the like. For example, the audio evaluation routine may include explicit hearing test routines, including traditional audiology, speech tests in noise, audio games, spatial resolution tests, listening efforts, localization accuracy, performance monitoring, and post-experience surveys. The audio evaluation routine generates audio evaluation data that describes the user's response to the audio evaluation routine. For example, the audio evaluation data may include binary data indicating whether the user reacts to the sound, an accuracy value indicating the degree to which the user understands speech in different noise environments, the duration and smoothness of directional movement toward the sound source, the ratio of front / back confusion, localization error, minimum audible angle, the degree of difference in binaural masking levels, and the like.

[0080] The personalization module 290 stores the audio evaluation data in an audio profile. In some embodiments, the audio profile may be stored locally on the wearable device, such as in the data storage 235. In some embodiments, the personalization module 290 may transmit the evaluation data or the audio profile to an external system, such as a social networking system. The personalization module 290 may continuously update the audio profile as additional evaluation data is captured.

[0081] In some embodiments, as part of the audio evaluation routine, the personalization module 290 obtains perceptual feedback from the user. Perceptual feedback can indicate whether the user responds to the sound, whether the user understands the speech in the audio content, or indicates the perceptual positioning of the synthesized sound. In some embodiments, perceptual feedback can include the gaze direction of the user's eyes, indicating that the user perceives the sound emanating from the gaze direction. Perceptual feedback can include a verbal response from the user, such as "front", "back", "left" or "right", or the user's repetition of a phrase presented to the user. Perceptual feedback can include the user's movement, such as the user turning their head or pointing a finger in a direction.

[0082] The personalization module 290 captures active and passive responses from the user. These responses can be captured by any sensor on the head-mounted device, such as through microphones, hand tracking, controllers, pupillary measurement, electroencephalogram (EEG), galvanic skin response (GSR), eye tracking modules, tactile feedback from gloves, position sensors (such as IMU) that capture the movement of the head-mounted device, etc. In some embodiments, sensory feedback can be captured by external sensors, such as cameras that detect the position of the head-mounted device or the user.

[0083] In some embodiments, sensory feedback may be obtained during an active assessment routine. For example, the headset may notify the user that an audio assessment is taking place, and the headset may provide audio and / or visual instructions to perform an action, such as repeating a phrase or looking in the direction of a sound.

[0084] In some embodiments, sensory feedback can be obtained during a passive assessment routine, where the user may not be aware that an audio assessment is being performed. For example, the user may interact with the head mounted device, such as participating in a virtual reality game, and the personalization module 290 may monitor the user's response to the sounds during the virtual reality game.

[0085] The personalization module 290 compares the perceived feedback of each sound source in the audio evaluation routine with one or more attributes of the sound source. The audio system can generate one or more sound sources for the audio evaluation routine. The sound source generated by the audio system may have known attributes. One or more sound sources may be generated by objects or people in a local area. The audio system can determine the attributes of the sound source, such as tracking positioning and measuring frequency, and compare the user response with the measured attributes. The attributes of the sound source can include positioning, amplitude, frequency, words in speech, etc.

[0086] The personalization module 290 may determine an accuracy value for one or more attributes of each sound source. For example, the personalization module 290 may assign a scalar accuracy value between 1-10, with 10 indicating a highly accurate response of the user to the sound source. Each sound source may include an accuracy value for each attribute. For example, a first accuracy value for a sound source may indicate how well the user understands the words in a phrase, and a second accuracy value may indicate a difference between the user's perceived location of the sound source and the actual or expected location of the sound source.

[0087] The personalization module 290 analyzes the results of the audio evaluation to create an audio profile. In some embodiments, the personalization module 290 analyzes the results locally on the head mounted device. In some embodiments, the personalization module 290 transmits the results to an external system, and the external system generates an audio profile.

[0088] The audio profile may describe the environmental conditions associated with the user's positive or negative evaluation data. For example, the environmental conditions may include the geometry of the room or local area, GPS coordinates, time of day, the language being spoken, signal-to-noise ratio, reverberation, background noise characteristics, etc. The audio profile may indicate whether the user is able or unable to understand speech under specific environmental conditions.

[0089] In addition, the audio profile may include results related to the user's physical features that affect hearing, such as personalized HRTFs and body-related transfer functions to generate a customized audio profile. The body data may describe the shape of the user's head, ears, and torso. The body data may be obtained from user pictures captured by a wearable device, or retrieved from a database, such as from a social networking system. The audio profile may include an HRTF that is partially generated based on the shape. The audio profile may use HRTF in conjunction with audio evaluation data to provide instructions for the user's subsequent sound presentation. For example, the HRTF of a sound source at a specific angle may attenuate certain frequencies more than others, which may prevent the user from hearing a certain frequency. However, based on the audio evaluation data, the audio profile may indicate that the frequency should be increased to a dB level that the user can hear. Therefore, by combining HRTF with audio evaluation data, the audio profile may contain a complete set of parameters for improving the user's audio experience.

[0090] In some embodiments, the personalization module 290 can store the audio profile locally on the head mounted device, for example in the data storage 235. In some embodiments, the personalization module 290 can transmit the audio profile to an external system, and the external system can store the audio profile in a database. For example, the audio profile can be stored in a database of a social network and associated with a user account.

[0091] In response to the audio controller 230 determining to present audio content to the user, the audio controller 230 queries the personalization module 290 for parameters in the audio profile. The personalization module 290 retrieves the audio profile of the user, for example, by querying the data store 235 or by querying an external system.

[0092] The personalization module 290 uses the audio profile to modify the sound presented to the user. Using the audio profile can improve the audio of all audio experiences of the user, including video calls, virtual reality games and applications, augmented reality telepresence, real-time voice enhancement, real-time noise reduction, etc. For example, during a video call, if the parameters in the audio profile indicate that the user has low sensitivity in certain frequency ranges, the personalization module 290 can increase the amplitude of the speaker speaking in these frequency ranges. For virtual reality games, if the parameters in the audio file indicate that the user has difficulty understanding speech in a reverberant environment, the personalization module 290 can reduce the amount of reverberation in the sound presented to the user. The personalization module 290 can provide parameters or instructions to the sound filter module 280 based on the parameters to adjust the sound filter according to the parameters.

[0093] In some embodiments, the personalization module 290 can store enhancement schemes associated with the audio profile in the data storage 235. An enhancement scheme is a set of predefined parameters for modifying the sound for the user. The enhancement scheme can be used to replace the parameters in the user's audio profile. Different users may have similar hearing characteristics, so that different users may benefit from similar modifications to the sound presented to the user. By grouping multiple users into subsets associated with the enhancement scheme, the system can reduce the complexity and processing power for generating enhanced audio experiences for the user. For example, in some embodiments, the system can include ten different enhancement schemes, or one thousand different enhancement schemes, and one of the enhancement schemes can be assigned to the audio profile based on the similarity between the user's evaluation data and the selected enhancement scheme.

[0094] In response to the audio system 200 detecting a new user, the audio system 200 may attempt to retrieve an audio profile for the new user. The audio system 200 may identify the new user based on, for example, login information, a password, facial recognition, etc. In some embodiments, the user may provide the audio system 200 with a social network identification of the user. The personalization module 290 may query the data storage 235 or an external system for an audio profile for the new user. If the data storage 235 or the external system identifies an audio profile for the new user, the data storage 235 or the external system may transmit the audio profile to the audio system 200. If the personalization module 290 cannot obtain an audio profile for the new user, the personalization module 290 may initiate an audio evaluation of the new user in order to generate an audio profile.

[0095] Systems and applications that present sound to the user can access the audio profile and modify the sound presented to the user based on the data in the audio profile to enhance the user's audio experience. The head-mounted device, external system, or third-party application can access the audio profile to generate sound for the user of the head-mounted device. For example, the user can use any device available to the user to access third-party audio content, such as a game application or website. The user can provide the user ID of the social network to the third party, and the third-party system can query the social network for the user's audio files. The third-party system can modify the audio content presented to the user based on the parameters in the audio file.

[0096] The audio profile may include security settings. The security settings may allow only authorized parties to access the audio profile, or only allow access to certain portions of the data stored in the audio profile. For example, the audio profile may be stored on a social network with the user's user profile, and the security settings may indicate that only entities associated with the user in the social network can access the user's audio profile. In some embodiments, the data in the audio profile may be encrypted and the encryption key may be stored on the user's wearable device, so that a third-party system can use the audio profile to present content to the user on the user's device without providing the third-party system with access to the audio profile content.

[0097] Figure 3 is a flow chart of a method 300 of generating enhanced audio content according to one or more embodiments. Figure 3 The process shown may be performed by components of an audio system (e.g., audio system 200). In other embodiments, other entities may perform Figure 3 Embodiments may include different and / or additional steps, or perform the steps in a different order.

[0098] The audio system on the wearable device collects 310 audio evaluation data of the user. Collecting the audio evaluation data may include performing an audio evaluation routine. The audio evaluation routine may include monitoring the user's response to the audio content. For example, the user may repeat a phrase, provide a rating for the audio content, look in the direction of a sound source, etc.

[0099] The audio system updates 320 an audio profile based on the audio evaluation data and the physical data describing the user. The audio profile may include parameters indicating how to adjust the audio content for presentation to the user. For example, based on the results of the audio evaluation routine, the parameters may indicate that the amplitude of certain frequency bands should be increased for the user, or that the reverberation level in the artificial audio environment should be reduced to allow the user to understand speech. The physical data may describe the shape of the user's head, ears, and torso. The physical data may be obtained from a picture of the user captured by a wearable device, or retrieved from a database, such as from a social networking system. The audio profile may include an HRTF generated in part based on the shape.

[0100] The audio system stores 330 the audio profile. In some embodiments, the audio system stores the audio profile locally on the wearable device. In other embodiments, the audio system can transmit the audio profile to an external system, such as a social media network, and the external system can store the audio profile with the user's user profile. The audio system or the external system can assign an enhancement scheme to the audio profile.

[0101] The audio system presents 340 audio content to the user based in part on the audio profile. The audio content may include audio for a virtual reality game, sounds produced by objects in the user's local area, music, or any other audio suitable for production by a wearable device. The audio system may modify the audio content based on parameters in the audio profile. In some embodiments, the audio system may modify the audio content based on an enhancement scheme associated with the audio file.

[0102] In some embodiments, the audio content can be presented to the user by any device, such as a device that the user has not used before. The device can request an audio profile from an external system, and the device can present customized audio content to the user based on the audio profile.

[0103] Figure 4 is a system 400 including a wearable device 405 according to one or more embodiments. In some embodiments, the wearable device 405 may be Figure 1A The head mounted device 100 or Figure 1B The head mounted device 105. The system 400 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 4 The illustrated system 400 includes a wearable device 405, an input / output (I / O) interface 410 coupled to a console 415, a network 420, a mapping server 425, and an external system 470. Figure 4 An example system 400 is shown that includes one wearable device 405 and one I / O interface 410, but in other embodiments, any number of these components may be included in the system 400. For example, there may be multiple head-mounted devices, each head-mounted device having an associated I / O interface 410, and each head-mounted device and I / O interface 410 communicating with a console 415. In alternative configurations, different and / or additional components may be included in the system 400. Furthermore, in some embodiments, in combination with Figure 4 The functionality described by one or more of the components shown may be combined with Figure 4 The described approaches are distributed among the components in different ways. For example, some or all of the functions of the console 415 can be provided by the wearable device 405.

[0104] Wearable device 405 includes display assembly 430, optical block 435, one or more position sensors 440, and DCA 445. Some embodiments of wearable device 405 have a Figure 4 In addition, in other embodiments, by combining Figure 4 The functionality provided by the various components described may be distributed variously among the components of wearable device 405 , or may be captured in separate components of wearable device 405 .

[0105] The display assembly 430 displays content to the user based on the data received from the console 415. The display assembly 430 displays content using one or more display elements (e.g., the display element 120). The display element can be, for example, an electronic display. In various embodiments, the display assembly 430 includes a single display element or multiple display elements (e.g., a display for each eye of the user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof. It should be noted that in some embodiments, the display element 120 can also include some or all of the functionality of the optical block 435.

[0106] The optical block 435 can amplify the image light received from the electronic display, correct the optical errors associated with the image light, and present the corrected image light to one or two windows of the wearable device 405. In various embodiments, the optical block 435 includes one or more optical elements. Example optical elements included in the optical block 435 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 435 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 435 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0107] The magnification and focusing of the image light by the optical block 435 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content is presented using almost all of the user's field of view (e.g., approximately 110 degrees diagonal), and in some cases all of the field of view. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0108] In some embodiments, optical block 435 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and when optical block 435 receives image light generated based on the content from the electronic display, optical block 435 corrects the distortion.

[0109] The position sensor 440 is an electronic device that generates data indicating the position of the wearable device 405. The position sensor 440 generates one or more measurement signals in response to the movement of the wearable device 405. The position sensor 190 is an embodiment of the position sensor 440. Examples of the position sensor 440 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, or some combination thereof. The position sensor 440 may include multiple accelerometers that measure translational movement (forward / backward, up / down, left / right) and multiple gyroscopes that measure rotational movement (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the wearable device 405 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate the velocity vector, and integrates the velocity vector over time to determine the estimated position of the reference point on the wearable device 405. A reference point is a point that can be used to describe the location of the wearable device 405. Although a reference point may generally be defined as a point in space; however, in practice, a reference point is defined as a point within the wearable device 405.

[0110] The DCA 445 generates depth information for a portion of the local area. The DCA includes one or more imaging devices and a DCA controller. The DCA 445 may also include an illuminator. The operation and structure of the DCA 445 are described above with respect to Figure 1A Described.

[0111] The audio system 450 provides audio content to the user of the wearable device 405. The audio system 450 may be an embodiment of the audio system 200 described above. The audio system 450 may include one or more acoustic sensors, one or more transducers, and an audio controller. The audio system 450 may provide spatialized audio content to the user. In some embodiments, the audio system 450 may request acoustic parameters from the mapping server 425 via the network 420. The acoustic parameters describe one or more acoustic characteristics (e.g., room impulse response, reverberation time, reverberation level, etc.) of a local area. The audio system 450 may provide information describing at least a portion of a local area from, for example, a DCA 445 and / or positioning information of the wearable device 405 from a position sensor 440. The audio system 450 may generate one or more sound filters using one or more acoustic parameters received from the mapping server 425, and use the sound filters to provide audio content to the user.

[0112] The audio system 450 may perform an audio assessment routine. The audio system 450 may create an audio profile 480 for the user based on the audio assessment routine. The audio system 450 may transmit the audio profile 480 and / or the results of the audio assessment routine to the external system 470, and the external system 470 may store the audio profile 480. In some embodiments, the audio system 450 may request the audio profile of the user from the external system 470. The audio system 450 may modify the audio content presented to the user based on the parameters stored in the audio profile.

[0113] The I / O interface 410 is a device that allows a user to send an action request and receive a response from the console 415. An action request is a request to perform a specific action. For example, an action request can be an instruction to start or end capturing image or video data, or an instruction to perform a specific action within an application. The I / O interface 410 may include one or more input devices. Example input devices include a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 415. The action request received by the I / O interface 410 is transmitted to the console 415, and the console 415 performs the action corresponding to the action request. In some embodiments, the I / O interface 410 includes an IMU that captures calibration data indicating an estimated position of the I / O interface 410 relative to the initial position of the I / O interface 410. In some embodiments, the I / O interface 410 can provide tactile feedback to the user according to the instruction received from the console 415. For example, tactile feedback is provided when an action request is received, or when console 415 transmits instructions to I / O interface 410 that cause I / O interface 410 to generate tactile feedback when console 415 performs the action.

[0114] The console 415 provides content to the wearable device 405 for processing based on information received from one or more of: the DCA 445, the wearable device 405, and the I / O interface 410. Figure 4 In the example shown, the console 415 includes an application store 455, a tracking module 460, and an engine 465. Some embodiments of the console 415 have a Figure 4 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Figure 4 The described approach is distributed among the components of console 415. In some embodiments, the functionality discussed herein with reference to console 415 can be implemented in wearable device 405 or in a remote system.

[0115] The application storage 455 stores one or more applications for execution by the console 415. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be in response to input received from a user via movement of the wearable device 405 or the I / O interface 410. Examples of applications include: gaming applications, conference applications, video playback applications, or other suitable applications. In some embodiments, one or more applications may be stored on an external server that communicates with the console 415 or the wearable device 405 via a network 420. The application may generate audio content for presentation on the wearable device 405, and the audio system 450 may modify the audio content based on the audio profile 485. In some embodiments, the application transmits the audio content to the audio system 450, and the audio system 450 modifies the audio content based on parameters in an audio file stored locally on the wearable device 405.

[0116] The tracking module 460 uses information from the DCA 445, one or more position sensors 440, or some combination thereof to track the movement of the wearable device 405 or the I / O interface 410. For example, the tracking module 460 determines the location of a reference point of the wearable device 405 in a map of the local area based on the information from the wearable device 405. The tracking module 460 can also determine the location of an object or a virtual object. In addition, in some embodiments, the tracking module 460 can use a portion of the data from the position sensor 440 indicating the location of the wearable device 405 and the representation of the local area from the DCA 445 to predict the future location of the wearable device 405. The tracking module 460 provides the estimated or predicted future location of the wearable device 405 or the I / O interface 410 to the engine 465.

[0117] The engine 465 executes the application and receives the position information, acceleration information, speed information, predicted future position or some combination thereof of the wearable device 405 from the tracking module 460. Based on the received information, the engine 465 determines the content to be provided to the wearable device 405 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 465 generates content for the wearable device 405, which reflects (mirror) the user's movement in a virtual local area or in a local area that enhances the local area with additional content. In addition, the engine 465 executes the action within the application executed on the console 415 in response to the action request received from the I / O interface 410, and provides feedback to the user that the action is executed. The feedback provided can be visual or auditory feedback via the wearable device 405 or tactile feedback via the I / O interface 410.

[0118] The network 420 couples the wearable device 405 and / or the console 415 to the mapping server 425. The network 420 may include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, the network 420 may include the Internet and a mobile phone network. In one embodiment, the network 420 uses standard communication technologies and / or protocols. Therefore, the network 420 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 420 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged over the network 420 may be represented using techniques and / or formats including image data in binary form (e.g., portable network graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or part of the links may be encrypted using conventional encryption techniques, such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.

[0119] The mapping server 425 may include a database storing a virtual model describing a plurality of spaces, wherein a location in the virtual model corresponds to a current configuration of a local area of ​​the wearable device 405. The mapping server 425 receives information describing at least a portion of the local area and / or location information of the local area from the wearable device 405 via the network 420. The mapping server 425 determines a location associated with the local area of ​​the wearable device 405 in the virtual model based on the received information and / or location information. The mapping server 425 determines (e.g., retrieves) one or more acoustic parameters associated with the local area based in part on the location determined in the virtual model and any acoustic parameters associated with the determined location. The mapping server 425 may transmit the location of the local area and any acoustic parameter values ​​associated with the local area to the wearable device 405.

[0120] The external system 470 may be a system that stores user profiles, such as a social network. The external system 470 may store a user profile 475 of a user of the wearable device 405. The user profile 475 may include information about the user, such as demographic information, content uploaded by the user, connections between the user and other users of the external system 470, etc. The external system 470 may store an audio profile 480 as part of or associated with the user profile 475. The audio profile 480 may contain audio evaluation data received from the wearable device 405.

[0121] In some embodiments, the external system 470 may store an enhancement scheme associated with the audio profile. An enhancement scheme is a set of predefined parameters for modifying the sound for the user. The enhancement scheme may be used to replace the parameters in the user's audio file.

[0122] Additional configuration information

[0123] The foregoing description of the embodiments of the present disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art may recognize that many modifications and variations are possible in light of the above disclosure.

[0124] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0125] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0126] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0127] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information produced by the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0128] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the inventive subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the disclosure, which is set forth in the appended claims.

Claims

1. A method for presenting personalized audio content, comprising: Collecting audio evaluation data from users through wearable devices; Wherein collecting the audio assessment data comprises performing an audio assessment routine, the audio assessment routine comprising: Speech in noise testing; and generating a synthesized sound and detecting a gaze direction of the user using an eye tracking unit in response to the synthesized sound, wherein the gaze direction indicates a perceived localization of the synthesized sound; and wherein the audio evaluation data includes an accuracy value indicating the degree to which the user understands speech in different noise environments; updating an audio profile based on the audio evaluation data and the physical data describing the user; and Audio content is presented to the user based in part on the audio profile.

2. The method of claim 1, further comprising transmitting the audio profile to a social network, wherein the social network associates the audio profile with a user profile; and wherein the social network associates the audio profile with an enhancement scheme. 3 . The method of claim 1 , further comprising creating, by the wearable device, the audio profile.

4. The method according to claim 1, further comprising: receiving the audio content from an external system; and The audio content is modified based on the audio profile.

5. The method according to claim 1, wherein: Collecting the audio evaluation data includes: presenting sound to the user; and A response of the user to the sound is detected.

6. The method according to claim 1, wherein: Collecting the audio evaluation data includes: Determining the location of a sound source in a local area; and A response of the user to the sound source is detected.

7. The method according to claim 1, wherein: The body data describes the shape of the user's ears. The method of claim 1 , further comprising retrieving the audio profile from a social network.

9. The method according to claim 1, wherein: Collecting the audio assessment data comprises executing an audio assessment routine to determine parameters describing characteristics of the user's hearing; wherein the parameters comprise at least one of a scaling factor or a maximum reverberation level.

10. The method according to claim 1, wherein: The audio profile includes a security setting, wherein the security setting indicates whether a third party is authorized to access the audio profile.

11. A non-transitory computer-readable storage medium comprising computer program code for causing a processor to: Collecting audio evaluation data from users through wearable devices; Wherein collecting the audio assessment data comprises performing an audio assessment routine, the audio assessment routine comprising: Speech in noise testing; as well as generating a synthesized sound and detecting a gaze direction of the user using an eye tracking unit in response to the synthesized sound, wherein the gaze direction indicates a perceived localization of the synthesized sound; and wherein the audio evaluation data includes an accuracy value indicating the degree to which the user understands speech in different noise environments; updating an audio profile based on the audio evaluation data and physical data describing the user; and Audio content is presented to the user based in part on the audio profile.

12. The non-transitory computer-readable storage medium of claim 11, further comprising computer program code for execution by the processor to transmit the audio profile to a social network, wherein the social network associates the audio profile with a user profile.

13. A wearable device comprising a non-transitory computer-readable storage medium and a processor, the non-transitory computer-readable storage medium containing computer program code for the processor to perform the following operations: collecting audio evaluation data of the user through the wearable device; Wherein collecting the audio assessment data comprises performing an audio assessment routine, the audio assessment routine comprising: Speech in noise testing; as well as generating a synthesized sound and detecting a gaze direction of the user using an eye tracking unit in response to the synthesized sound, wherein the gaze direction indicates a perceived localization of the synthesized sound; and wherein the audio evaluation data includes an accuracy value indicating the degree to which the user understands speech in different noise environments; updating an audio profile based on the audio evaluation data and physical data describing the user; and Audio content is presented to the user based in part on the audio profile.

14. The wearable device of claim 13, further comprising computer program code for execution by the processor to transmit the audio profile to a social network, wherein the social network associates the audio profile with a user profile; and wherein the social network associates the audio profile with an enhancement scheme.

15. The wearable device of claim 13, further comprising computer program code for the processor to: receiving the audio content from an external system; and The audio content is modified based on the audio profile.

Images