Personalize equalization of audio output using a 3D reconstruction of the user's ears

Abstract

A method for generating a personalized audio output response for a head mounted device based on a representation of a user's ears. One or more images of a portion of a user's head including at least the user's ears are received. A representation of the user's ears is generated based in part on the one or more images. A simulation of sound propagation from an audio source to the user's ears is performed based on the representation. A personalized audio output response is generated for the user based on the simulation, the personalized audio output response being configured to adjust one or more acoustic parameters of audio content provided to the user by the head mounted device.

Description

Field of the Invention

[0001] The present disclosure relates generally to artificial reality systems and, more particularly, to generating personalized equalization filters for users.

[0002] background

[0003] Existing head-mounted devices, such as artificial reality (AR) and virtual reality (VR) headsets, typically use multiple transducers to provide audio content to a user. However, the propagation of sound from the transducers to the user's ears may vary based on the anatomical features of the user's ears and / or head. For example, differences in ear size and shape between users can affect the sound generated by the head-mounted device and the sound perceived by the user, which can negatively impact the user's audio experience. Current audio systems are insufficient to deliver high-fidelity audio content because they may not take into account variations in anatomical features between users and inconsistencies in the fitment of the head-mounted device between users. Therefore, there is a need for a method for adjusting audio output that compensates for variations in anatomical features and inconsistencies in fitment so that the audio content provided by the head-mounted device is customized to the user.

[0004] Overview

[0005] A system and method for generating a personalized equalization filter based on an image of a user's ear to enhance the user's audio experience is disclosed. A representation of at least a portion of a user's head including at least the user's ear is generated using one or more images of the user's head. The image may include an image of the user (e.g., the user's head, the user's ear) and / or an image of the user wearing a head-mounted device. The head-mounted device may include multiple transducers for providing audio content to the user. The representation may be a three-dimensional mesh model. A simulation is performed based on the representation, which numerically simulates how sound propagates from an audio source (e.g., a transducer array of a head-mounted device) to the user's ear. The simulation is used to generate an equalization filter for the user. The equalization filter is then used to adjust one or more acoustic parameters (e.g., wavelength, frequency, volume, pitch, balance, etc.) of the user's audio output to generate a target response at the user's ear so that the user perceives the audio output in the way the creator of the audio output intended it to be heard. The equalization filter may also be attached to the user's social network profile.

[0006] According to some embodiments of the present invention, a method is provided, comprising: receiving an image including an ear of a user; generating a three-dimensional (3D) representation of the ear of the user based in part on the received image; performing a simulation of sound propagation from an audio source to the ear of the user based on the 3D representation; and generating an equalization filter for the user based on the simulation, the equalization filter being configured to adjust one or more acoustic parameters of audio content provided to the user.

[0007] In some embodiments, in the image of the user's ear, the user is wearing a head mounted device, and wherein generating the 3D representation of the user's ear also includes generating a 3D representation including the head mounted device.

[0008] In some embodiments, the equalization filter, when applied to audio content provided to a user, adjusts one or more acoustic parameters of the audio content for the user based on a simulation of sound propagation from an audio source to an ear of the user.

[0009] In some embodiments, the audio source is a speaker array of a head mounted device.

[0010] In some embodiments, the simulation of sound propagation from the audio source to the user's ear simulates the audio output at the user's ear.

[0011] In some embodiments, generating the 3D representation includes processing an image of the user's ear using a machine learning model to obtain the 3D representation.

[0012] In some embodiments, the 3D representation is generated using a principal component analysis (PCA) model that describes the user's ear as a combination of representative three-dimensional shapes of the test subject's ears.

[0013] In some embodiments, a PCA model is generated by the following steps: receiving an ear image of a test subject and a measured audio output response of the test subject; determining a PCA-based 3D representation of the ear image of the test subject based on an initial PCA model; simulating the PCA-based 3D representation to determine a simulated audio output response of the test subject; determining a difference between the simulated audio output response and the measured audio output response; generating an updated PCA model based on the determined difference by modifying the initial PCA model or by updating one or more intermediate PCA models derived from the initial PCA model; and when the determined difference is below a threshold, determining the updated PCA model as the PCA model for representing the user's ear.

[0014] In some embodiments, the method further comprises training a machine learning model based on the PCA model using the test subject's ear image and the PCA-based 3D representation of the test subject's ear image.

[0015] In some embodiments, images of the user's ears are captured by a depth camera component of the head mounted device.

[0016] In some embodiments, the generated 3D representation includes a three-dimensional mesh of the user's ear.

[0017] In some embodiments, the method further includes providing the generated equalization filter to a head mounted device, the head mounted device being configured to provide audio content to a user using the equalization filter.

[0018] In some embodiments, the method further comprises providing the generated equalization filter to an online system, wherein the online system stores the generated equalization filter in association with the online profile of the user.

[0019] In some embodiments of the present invention, a non-transitory computer-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to perform steps including the following steps: receiving an image including a user's ear; generating a 3D representation of the user's ear based in part on the received image; performing a simulation of sound propagation from an audio source to the user's ear based on the 3D representation; and generating an equalization filter for the user based on the simulation, the equalization filter adjusting one or more acoustic parameters of audio content provided to the user.

[0020] In some embodiments, in the image of the user's ear, the user is wearing a head mounted device, and wherein generating the 3D representation of the user's ear also includes generating a 3D representation of the head mounted device.

[0021] In some embodiments, the equalization filter, when applied to audio content provided to a user, adjusts one or more acoustic parameters of the audio content for the user based on a simulation of sound propagation from an audio source to an ear of the user.

[0022] In some embodiments, the audio source is a speaker array of a head mounted device.

[0023] In some embodiments, the simulation is based on one of a finite element method (FEM), a boundary element method (BEM), and a finite difference time domain (FDTD) method, and wherein the simulation of sound propagation from an audio source to a user's ear simulates the audio output at the user's ear.

[0024] In some embodiments, the steps further include providing the generated equalization filter to a head mounted device, the head mounted device being configured to provide audio content to a user using the equalization filter.

[0025] In some embodiments, the steps further include providing the generated equalization filter to an online system configured to append the generated equalization filter to an online profile of the user.

[0026] It will be appreciated that any feature described herein as being suitable for one or more aspects or embodiments of the present invention is intended to be generalizable to any and all aspects and embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1A is a perspective view of a first embodiment of a head mounted device in accordance with one or more embodiments.

[0029] Figure 1Bis a perspective view of a second embodiment of a head mounted device according to one or more embodiments.

[0030] Figure 2 A system environment for providing audio content to a device in accordance with one or more embodiments is shown.

[0031] Figure 3 An equalization system in accordance with one or more embodiments is shown.

[0032] Figure 4A is an example view of an imaging device capturing an image of a user's head according to one or more embodiments.

[0033] Figure 4B According to one or more embodiments, Figure 4A An image of a portion of a user's head captured by an imaging device in the image processing unit.

[0034] Figure 5A is an example view of an imaging device capturing an image of a user's head wearing a head mounted device according to one or more embodiments.

[0035] Figure 5B According to one or more embodiments, Figure 5A An image of a portion of a user's head captured by an imaging device in the image processing unit.

[0036] Fig. 6A is an example view of an imaging device capturing an image of a user's head wearing a head mounted device with visual markers according to one or more embodiments.

[0037] Figure 6B According to one or more embodiments, Fig. 6A An image of a portion of a user's head captured by an imaging device in the image processing unit.

[0038] Figure 7 A method of generating a personalized equalization filter for a user based on simulations according to one or more embodiments is shown.

[0039] Fig. 8A An example process is shown for generating a representation of a user's ear using a machine learning model in accordance with one or more embodiments.

[0040] Figure 8B is a flow chart for determining a PCA model according to one or more embodiments.

[0041] Fig. 9A A machine learning model for predicting audio output at a user's ear is shown in accordance with one or more embodiments.

[0042] Fig. 9BA method of generating a personalized equalization filter using a machine learning model according to one or more embodiments is shown.

[0043] Fig.10 is a block diagram of an audio system according to one or more embodiments.

[0044] Fig.11 A system environment for providing audio content to a user according to one embodiment.

[0045] The accompanying drawings depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0046] Detailed Description

[0047] Overview

[0048] A head mounted device such as an artificial reality (AR) head mounted device includes one or more transducers (e.g., speakers) for providing audio content to a user. However, the sound propagation from the transducer to the user's ear may vary from user to user and device to device. In particular, the audio output at the user's ear may vary based on the anthropometric features of the user's ear and / or head. Anthropometric features are physical characteristics of the user (e.g., ear shape, ear size, orientation / position of the ear on the head, head size, etc.). In addition, the fit of the head mounted device may vary due to anthropometric features, which may also affect the audio output response. In this way, it may be useful to adjust the audio content provided to the user by the head mounted device so that the user experiences a personalized audio output response, thereby enhancing the user experience and providing high-quality content to the user. Therefore, an equalization filter that adjusts one or more acoustic parameters of the audio output (e.g., wavelength, frequency, volume, pitch, balance, other spectral content, acoustic time delay, etc.) is generated based on the user's ear. When applied to audio content, the equalization filter adjusts the audio content to the target response at the user's ear so that the user perceives the audio content in the way the creator of the audio content intends for it to be heard. In one embodiment, the target response is associated with a predetermined value (or a range of acceptable values) for each parameter in the set of acoustic parameters. The predetermined value (or range of acceptable values) for each parameter in the set of acoustic parameters corresponds to a relatively high acceptable threshold of the sound quality that the content creator wants the audio content to be perceived by the user.

[0049] In one embodiment, an imaging system (e.g., a user's mobile device, etc.) captures one or more images of a user of a head-mounted device to collect anthropometric information associated with the user. The imaging system may capture image data (e.g., still image data or video image data) of a user's ear, a user's head, and / or a user wearing a head-mounted device. In one embodiment, each of the one or more images is a frame in a captured video of a user's ear, a user's head, and / or a user wearing a head-mounted device. The head-mounted device may be a virtual reality (VR) head-mounted device, an AR head-mounted device, or some other head-mounted device configured to provide audio content to a user. The head-mounted device may include multiple transducers for providing audio content, and the positions of the transducers may be known. The size of the head-mounted device may also be known. In some embodiments, the head-mounted device includes one or more visual markers for determining position information relative to the user's head. For example, the head-mounted device may include markers positioned along its frame (e.g., along each temple arm). The position of each marker relative to other markers and the head-mounted device is known. In some embodiments, each marker has a unique size and / or shape.

[0050] The equalization system receives one or more images of the user (e.g., from an imaging system, from a head-mounted device, etc.) to generate a customized equalization filter for the user. In one embodiment, the imaging system provides one or more images to the head-mounted device, and the head-mounted device provides the one or more images to the equalization system. The equalization system identifies features (e.g., shape, size) of the user's ears based on the received images. In some embodiments, the equalization system extracts depth information associated with the image and generates a 3D representation of the user's ears based on the extracted depth information and the identified features. The equalization system can generate the 3D representation using a machine learning model, and in some embodiments, the 3D representation includes a representation of the head-mounted device. The equalization system performs a simulation of audio propagation from an audio source (e.g., a transducer array of the head-mounted device) to the 3D representation of the user's ears. Based on the simulation, the equalization system can predict the audio output at the user's ears. Based on the difference between the target audio response and the predicted audio output at the user's ears, an equalization filter is generated for the user. In one embodiment, the equalization filter is generated based on a transfer function that is a ratio between two complex frequency responses (i.e., the target response and the predicted response). The equalization filter adjusts one or more acoustic parameters of the audio output (e.g., wavelength, frequency, volume, pitch, balance, other spectral content, acoustic time delay, etc.) based on the user's ears to generate a target response at the user's ears so that the user perceives the audio output as the creator of the audio output intended it to be heard. In some embodiments, the equalization system generates an audio profile for the user that specifies an amount of compensation for one or more acoustic parameters based on the equalization filter.

[0051] In another embodiment, the equalization system uses a machine learning model to predict the audio output at the user's ear. The equalization system receives one or more images (e.g., received from an imaging system) and extracts one or more features describing the user's ear based on the image. The equalization system can use machine learning techniques, imaging techniques, algorithms, or any other model to extract features of the user's ear based on the image. The equalization system uses a machine-learned model to determine the audio output at the user's ear based on the extracted one or more features. In one embodiment, the model is trained using images of other user ears / heads with previously identified features (e.g., identified by the model, identified by a person) and known audio output at each user's ear. Based on the difference between the target audio response and the predicted audio output at the user's ear, an equalization filter is generated for the user. The equalization filter adjusts one or more acoustic parameters of the audio output based on the user's ear (e.g., wavelength, frequency, volume, pitch, balance, other spectral content, acoustic time delay, etc.) to generate a target response at the user's ear so that the user perceives the audio output in the way the creator of the audio output intends it to be heard.

[0052] The equalization system can provide the generated personalized equalization filter to the head-mounted device. In this way, the personalized equalization filter can modify one or more acoustic parameters of the audio content provided to the user by the head-mounted device so that the audio content is customized to the user. The personalized equalization filter improves the audio experience by reducing the audio output changes caused by differences between the user and the device. In addition, the personalized equalization filter can be attached to the user's profile (e.g., a social network profile) so that the user does not need to re-calibrate the device during subsequent use.

[0053] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used to create content in artificial reality and / or be used in other ways in artificial reality. Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted devices) connected to a host computer system, standalone wearable devices (e.g., head-mounted devices), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0054] Example Headset

[0055] Figure 1A is a perspective view of a first embodiment of a head mounted device according to one or more embodiments. In some embodiments, the head mounted device is a near eye display (NED) or an eyewear device. Typically, the head mounted device 100 can be worn on a user's face so that content (e.g., media content) is presented using a display component and / or an audio system. However, the head mounted device 100 can also be used so that the media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 100 include one or more images, videos, audio, or some combination thereof. The head mounted device 100 includes a frame and may include, among other components, a display component that includes one or more display elements 120, a depth camera assembly (DCA), an audio system, and a position sensor 190. Although Figure 1A Components of the head mounted device 100 are shown at example locations on the head mounted device 100, but the components may be located elsewhere on the head mounted device 100, on a peripheral device that is paired with the head mounted device 100, or some combination of both. Similarly, components on the head mounted device 100 may be located more conveniently than on the head mounted device 100. Figure 1A More or less than shown.

[0056] The frame 110 holds the other components of the head mounted device 100. The frame 110 includes a front portion that holds one or more display elements 120 and end pieces (e.g., temples) that attach to the user's head. The front portion of the frame 110 bridges the top of the user's nose. The length of the end pieces can be adjustable (e.g., adjustable temple lengths) to fit different users. The end pieces can also include portions that curl behind the user's ears (e.g., temple tips, ear hooks). In some embodiments, the frame 110 includes one or more visual markers, which are described below with respect to Figure 6A-6B Described in more detail.

[0057] One or more display elements 120 provide light to a user wearing the head mounted device 100. As shown, the head mounted device includes a display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light that is provided to the eyebox of the head mounted device 100. The eyebox is the spatial location occupied by the eye when the user wears the head mounted device 100. For example, the display element 120 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional light source, one or more linear light sources, one or more point light sources, etc.) and one or more waveguides. Light from the light source is coupled inwardly into one or more waveguides, which output light in such a way that there is a pupil replication in the eyebox of the head mounted device 100. The inward coupling and / or outward coupling of light from one or more waveguides can be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a reflector, etc.) that scans the light from the light source as it is coupled inwardly into one or more waveguides. Note that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a local area around the head mounted device 100. The local area is an area around the head mounted device 100. For example, the local area may be a room where a user wearing the head mounted device 100 is located, or the user wearing the head mounted device 100 may be outside and the local area is an external area. In this case, the head mounted device 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent so that light from the local area can be combined with light from one or more display elements to produce AR and / or MR content.

[0058] In some embodiments, display elements 120 do not generate image light, but rather lenses transmit light from localized areas to the viewport. For example, one or both of display elements 120 may be non-corrective (non-prescription) lenses, or prescription lenses (e.g., single vision lenses, bifocal and trifocal lenses, or progressive lenses) to help correct for imperfections in the user's vision. In some embodiments, display elements 120 may be polarized and / or tinted to protect the user's eyes from sunlight.

[0059] It should be noted that in some embodiments, the display element 120 may include an additional optical block (not shown). The optical block may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that direct light from the display element 120 to the viewing window. The optical block may, for example, correct aberrations in some or all image content, magnify some or all images, or some combination thereof.

[0060] The DCA determines depth information of a portion of a local area around the head mounted device 100. The DCA includes one or more imaging devices 130 and a DCA controller ( Figure 1A 140). In some embodiments, the illuminator 140 illuminates a portion of the local area with light. The light may be, for example, structured light (e.g., dot patterns, stripes, etc.) in infrared (IR), IR flash for time of flight, etc. In some embodiments, one or more imaging devices 130 capture an image of a portion of the local area including light from the illuminator 140. As shown in the figure, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, there is no illuminator 140 and at least two imaging devices 130.

[0061] The DCA controller calculates depth information for the portion of the local area using the captured images and one or more depth determination techniques. The depth determination techniques may be, for example, direct time of flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from illuminator 140), some other technique for determining scene depth, or some combination thereof.

[0062] The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller 150. However, in other embodiments, the audio system may include different and / or additional components. Similarly, in some cases, the functions described with reference to the components of the audio system may be distributed between the components in a manner different from that described herein. For example, some or all of the functions of the audio controller may be performed by a remote server.

[0063] The transducer array presents sound to the user. The transducer array includes multiple transducers. The transducer can be a speaker 160 or a tissue transducer 170 (e.g., a bone conduction transducer or a cartilage conduction transducer). Although the speaker 160 is shown outside the frame 110, the speaker 160 can be enclosed in the frame 110. In some embodiments, instead of a separate speaker for each ear, the head mounted device 100 includes a speaker array that includes multiple speakers integrated into the frame 110 to improve the directionality of the presented audio content. The tissue transducer 170 is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to generate sound. The number and / or position of the transducers can be different from the head mounted device 100. Figure 1A Different than shown.

[0064] The sensor array detects sounds in a local area of ​​the head mounted device 100. The sensor array includes a plurality of acoustic sensors 180. The acoustic sensors 180 capture sounds emitted from one or more sound sources in a local area (e.g., a room). Each acoustic sensor is configured to detect sounds and convert the detected sounds into an electronic format (analog or digital). The acoustic sensor 180 may be a sound wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sounds.

[0065] In some embodiments, one or more acoustic sensors 180 can be placed in the ear canal of each ear (e.g., acting as binaural microphones). In some embodiments, acoustic sensors 180 can be placed on an exterior surface of head mounted device 100, on an interior surface of head mounted device 100, separate from head mounted device 100 (e.g., as part of some other device), or some combination thereof. The number and / or location of acoustic sensors 180 can vary depending on the device. Figure 1A For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected and to improve the sensitivity and / or accuracy of the information. The acoustic detection locations can be oriented so that the microphone can detect sound in a wide range of directions around the user wearing the head mounted device 100.

[0066] The audio controller 150 processes information from the sensor array describing the sounds detected by the sensor array. The audio controller 150 may include a processor and a computer-readable storage medium. The audio controller 150 may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and / or a head-related transfer function), track the location of a sound source, form a beam in the direction of a sound source, classify a sound source, generate a sound filter for the speaker 160, or some combination thereof.

[0067] The position sensor 190 generates one or more measurement signals in response to the movement of the head mounted device 100. The position sensor 190 may be located on a portion of the frame 110 of the head mounted device 100. The position sensor 190 may include an inertial measurement unit (IMU). Examples of the position sensor 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor 190 may be located outside the IMU, inside the IMU, or some combination thereof.

[0068] In some embodiments, the head mounted device 100 can provide simultaneous localization and mapping (SLAM) of the position of the head mounted device 100 and updates of the local area model. For example, the head mounted device 100 may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more red, green, and blue (RGB) cameras for capturing images of part or all of the local area. In some embodiments, some or all of the imaging devices 130 of the DCA may also be used as the PCA. The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the position sensor 190 tracks the position (e.g., positioning and posture) of the head mounted device 100 in the room. The following is combined with Fig.10 Additional details regarding components of head mounted device 100 are discussed.

[0069] Figure 1B 1 is a perspective view of a second embodiment of a head mounted device implemented as a head mounted display (HMD) according to one or more embodiments. In embodiments describing an AR system and / or an MR system, some portions of the front side of the HMD are at least partially transparent in the visible light band (approximately 380 nm to 750 nm), and portions of the HMD between the front side of the HMD and the user's eyes are at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a strap 175. The head mounted device 105 includes the same as described above with reference to FIG. Figure 1A Many of the same components are described, but modified to integrate with the HMD form factor. For example, the HMD includes a display assembly, a DCA, an audio system, and a position sensor 190. Figure 1B An illuminator 140 , a plurality of speakers 160 , a plurality of imaging devices 130 , a plurality of acoustic sensors 180 , and a position sensor 190 are shown.

[0070] System environment for delivering personalized audio content

[0071] Figure 2A system environment for providing personalized audio content to a user via a head mounted device according to one or more embodiments is shown. The system environment 200 includes a head mounted device 210, an imaging system 220, an equalization system 230, and an online system 240 connected via a network 250. The system environment 200 may include fewer or more components than described herein. In addition, the structure and / or function of the components may be different from those described herein.

[0072] The head mounted device 210 is a device configured to be worn by a user on the user's head region (e.g., head mounted device 100, head mounted device 105). The head mounted device 210 includes an audio system 215, which is configured to transmit audio content to the user wearing the head mounted device 210. The audio system 215 may include one or more transducers (e.g., speakers) for providing audio content to the user. Fig.10 The audio system 215 is described in more detail. In some embodiments, the head mounted device 210 includes additional components (e.g., a display system, a tactile feedback system) for providing other types of content (e.g., digital content, tactile content) to the user. In addition, the head mounted device 210 may include one or more visual markers for determining the position of the head mounted device 210 relative to the user wearing the device. The markers may be located along a frame of the head mounted device 210 (e.g., frame 110). The position of the markers relative to other markers and the head mounted device 210 is known. These markers are described below with respect to Figure 6A-6B Described in more detail.

[0073] The imaging system 220 includes an imaging device 225 that is configured to capture one or more images of a user's head, the head mounted device 210, and / or at least a portion of the user wearing the head mounted device 210. The imaging device 225 can be any suitable type of sensor, such as a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a hyperspectral imaging system, a LIDAR system (light detection and ranging system), a DCA, a dynamometer, an IR camera, some other imaging device, or some combination thereof. In this way, the imaging device 225 can capture RGB images, depth images (e.g., 3D images captured using a structured light camera, a stereo camera, etc.), or some other suitable type of image. In one embodiment, the imaging device 225 is a user device (e.g., a smartphone, a tablet, a laptop) having image capture capabilities. The imaging device 225 can additionally or alternatively capture video. Although in Figure 2, the imaging system 220 is shown as being separate from the head mounted device 210, but in alternative embodiments, the imaging system 220 is included in the head mounted device 210. For example, the imaging device 225 can be a camera coupled to the head mounted device 210 or a camera integrated in the head mounted device 210 (e.g., the imaging device 130).

[0074] In some embodiments, the imaging system 220 may apply one or more imaging techniques (e.g., stereo triangulation, light sheet triangulation, structured light analysis, time of flight analysis, interferometry) to determine depth information associated with an image captured by the imaging device 225. In a specific embodiment, the imaging system 220 includes a DCA that captures an image of the user, and the DCA uses the captured image to determine depth information of the user's head. The depth information describes the distance between the surface in the captured image and the DCA. The DCA may use one or more of stereo vision, photometric stereo, time of flight (ToF), and structured light (SL) to determine the depth information. The DCA may calculate the depth information from the captured image, or send the captured image to another component (e.g., the equalization system 230) to extract the depth information. In embodiments where the imaging system 220 does not include a DCA, the imaging system 220 may provide the captured image to the equalization system 230 or some other device and / or console to determine the depth information.

[0075] The equalization system 230 generates an equalization filter for the user, which adjusts one or more acoustic parameters of the audio content provided to the user via the head mounted device 210 so that the audio output of the head mounted device 210 matches the target response at the user's ear. In one embodiment, the equalization filter is generated based on the difference between the audio output at the entry ear point (EEP) or the ear drum reference point (DRP) and the target response. In this embodiment, the EEP refers to the entrance position of the ear canal, and the DRP refers to the position of the ear drum. The target response and its physically defined position may vary depending on the type of audio material presented. In one embodiment, the target response can be a flat frequency response measured at the EEP. In one embodiment, the equalization filter is generated based on a transfer function that is a ratio between two complex frequency responses (i.e., the target response and the predicted response).

[0076] Thus, the equalization filter adjusts the audio output based on the user's ears so that the user hears the audio output in the way the content creator intended the audio output to be heard. Figure 2, the equalization system 230 is shown as being separate from the head mounted device 210, but in some embodiments, the equalization system 230 may be included in the head mounted device 210. In some embodiments, the equalization system 230 generates a representation of at least a portion of the user's head (e.g., the ear) based on the image and / or video received from the imaging system 220. The equalization system 230 can simulate the audio output at the user's ear (e.g., the audio output from the head mounted device 210) based on the representation, and determine an equalization filter for the user based on the difference between the audio output at the user's ear and the target response. The target response is the way the content creator intends the user to hear the sound, which would be standard if there were no differences in ear shape and accessories. The target response is an ideal version of the audio output with the highest achievable sound quality. Therefore, the equalization filter specifies an amount of compensation for one or more acoustic parameters of the audio content provided to the user via the head mounted device 210 to account for differences in ear shape and accessories so that the user hears a version of the audio output that is as close to the target response as reasonably possible. In the following description of Figure 3 The audio system is described in more detail.

[0077] The online system 240 maintains user profile information and content to be presented to the user. For example, the online system 240 may be a social networking system. In some embodiments, the online system 240 stores a profile of a user of the head mounted device 210. In this way, the equalization system 230 may transmit an audio profile including one or more equalization filters to the online system 240, and the online system 240 may store the audio profile with the equalization filter together with the online profile of the user. The online system 140 may store personalized equalization filters for one or more devices corresponding to a single user. For example, the online system 140 may store a personalized equalization filter for the head mounted device 100 and another personalized equalization filter for the head mounted device 105 for the user. Therefore, the head mounted device 100 and the head mounted device 105 may retrieve the equalization filter of each device, and the head mounted device 100 and the head mounted device 105 may retrieve and use the equalization filter when providing content to the user. In this way, the user can use the head mounted device 210 without re-performing the process for generating a personalized equalization filter.

[0078] Network 250 may be any suitable communication network for data transmission. Network 250 is typically the Internet, but may be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network. In some example embodiments, network 250 is the Internet, and standard communication technologies and / or protocols are used. Thus, network 250 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Digital Subscriber Lines (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, PCI express advanced switching, etc. In some example embodiments, entities use customized and / or dedicated data communication technologies to replace or supplement the above technologies.

[0079] Balanced system

[0080] As described above, the equalization system 230 is configured to generate a personalized equalization filter for a user of the head mounted device 210. The equalization system 230 includes an image analysis module 305 and an audio customization model 325. In other embodiments, the equalization system 230 may include fewer or more components than described herein. In addition, the distribution of the functions of the components may be different from that described below.

[0081] The image analysis module 305 includes a feature extraction module 310, which is configured to extract information from one or more images of the user's head and / or ears captured by the user. The feature extraction module 310 receives an image from one or more components of the system environment 200 (e.g., the head-mounted device 210, the imaging system 220). The image can be a portion of the user's head (e.g., an ear), a portion of the user's head when the user wears a head-mounted device (e.g., the head-mounted device 100) or a head-mounted display (e.g., the head-mounted device 105). The feature extraction module 310 can extract information (e.g., depth information, color information) from the image and apply one or more techniques and / or models to determine features (e.g., size, shape) describing the user's ears and / or head. Examples include range imaging techniques, machine learning models (e.g., feature recognition models), algorithms, etc. In one embodiment, the feature extraction module 310 extracts anthropometric features that describe the user's physical characteristics (e.g., ear size, ear shape, head size, etc.).

[0082] In some embodiments, a machine learning model is used to train the feature extraction module 310. The feature extraction module 310 can be trained using images of other users with previously identified features. For example, multiple images can be labeled (e.g., by a person, by another model) with identified features of the user's ears and / or head (e.g., earlobe size and shape, ear position on the head, etc.). The image analysis module 305 can train the feature extraction module 310 using the images and associated features.

[0083] The image analysis module 305 further includes a depth map generator 315, which is configured to generate one or more depth maps based on the information (e.g., depth information) extracted by the feature extraction module 310. The depth map generator 315 can create a depth map of at least a portion of the user's head and identify the relative positions of the user's features. The depth map indicates the position or spatial relationship between the features of interest (e.g., ears) based on the image of the user's head. For example, the depth map can indicate the distance between the user's left ear and the right ear or the position of the user's ears relative to other features such as eyes and shoulders. In a similar manner, the depth map generator 315 can be used to create a depth map of the user's head wearing the head-mounted device based on the image of the head wearing the head-mounted device. In some embodiments, the depth map generator 315 can be used to create a depth map of the head of the user wearing the head-mounted device using a separately received head-mounted device image (i.e., not worn by the user).

[0084] The reconstruction module 320 generates a 3D representation of at least a portion of the user's head based on the features extracted by the feature extraction module 310 and / or the depth map generated by the depth map generator 315. More specifically, the reconstruction module 320 can generate a representation of one or both ears of the user. In one example, the reconstruction module 320 generates a representation of one ear (e.g., the left ear) and generates a mirror representation of the other ear (e.g., the right ear). Additionally or alternatively, the reconstruction module 320 can generate a three-dimensional mesh representation of the user's head, which describes, for example, the location of the user's head features, such as the location of the eyes, ears, neck, and shoulders. The reconstruction module 320 can combine the features of the user's head with the features of the head-mounted device 210 to obtain a representation of the user's head wearing the head-mounted device. In some embodiments, the representation of the head-mounted device 210 can be predetermined because the head-mounted device 210 worn by the user can have a unique known identifier to identify the device. In some embodiments, the head-mounted device 210 worn by the user can be identified from an image of the device taken using the imaging device 225 when it is worn.

[0085] In some embodiments, the reconstruction module 320 generates a PCA-based representation of the user based on an image of the head of a human test subject wearing a test headset and the measured audio output at the ear of each test subject. In the PCA-based representation, the user's head or features of the user's head (e.g., ear shape) are represented as a linear combination of principal components multiplied by corresponding PCA coefficients. To this end, the reconstruction module 320 receives an image and measured audio output from a test transducer set (e.g., a speaker array) at the user's ear, for example, received from a database. Based on the received images of the test subjects (e.g., 500-215 test subjects), the reconstruction module 320 performs a principal component analysis (PCA) that uses an orthogonal transform to determine a set of linearly uncorrelated principal components. For example, the orientation of the head-mounted device on the test subject's ear may be the focus of the PCA.

[0086] The reconstruction module 320 can generate a PCA model to determine the PCA-based geometry, which is described below with reference to Figures 8A-8B Although the PCA model is described as being generated and executed in the equalization system 230, the PCA model can be executed on a separate computing device. In this case, the results of the PCA are processed and provided to the reconstruction module 320 to process the PCA-based representation of the user.

[0087] The audio customization model 325 is configured to predict the audio output at the user's ear and generate a personalized equalization filter for the user based on the difference between the audio output at the ear and the target response. The audio customization model 325 includes a sound simulation module 330, an audio prediction module 335, and an equalization filter generator 345. In other embodiments, the audio customization model 325 may include additional components not described herein.

[0088] The sound simulation module 330 uses the representation generated by the reconstruction module 320 to simulate the audio output from the audio source (e.g., a speaker, a speaker array, a transducer of a head-mounted device, etc.) at the user's ear. In one example, the sound simulation module 330 generates a simulated audio output at the user's ear based on a representation of at least a portion of the user's head. In another example, the sound simulation module 330 generates a simulated audio output at the user's ear based on a representation of at least a portion of the user's head wearing the head-mounted device 210 (e.g., head-mounted device 100, head-mounted device 105). In addition, the head-mounted device 210 in the representation may include multiple transducers (e.g., speakers), and for the transducers in the representation (or a subset thereof), the sound simulation module 330 simulates the propagation of sound from the transducers to the user's ears. The sound simulation module 330 can also simulate the audio output at one or both ears of the user.

[0089] In one embodiment, the sound simulation module 330 is a numerical simulation engine. In order to obtain the simulated audio output at the ear, the sound simulation module 330 can use various simulation schemes, such as (i) the boundary element method (BEM), which is recorded in, for example, Carlos A. Brebbia et al. "Boundary Element Methods in Acoustics" (Springer, 1st edition, ISBN 1851666796 (1991)) and Gumerov NA et al. "A broadband fast multipole accelerated boundary element method for the three dimensional Helmholtzequation" (J. Acoust. Soc. Am., Vol. 125, No. 1, pp. 191-205 (2009)), (ii) the finite element method (FEM), which is recorded in, for example, Thompson, LL "A review of finite-element methods for time-harmonic acoustics" (J. Acoust. Soc. Am., Vol. 119, No. 3, pp. 1315-1330 (2006)), (iii) the finite difference time domain (FDTD) method, which is described, for example, in Taflove, A. et al., "Computational Electrodynamics: The Finite-Difference Time-Domain Method" (Artech House Press (2005), 3rd edition, Chapter 1.4) and Yee, K., "Numerical solution of initial boundary value problems involving Maxwell's equations in isotropic media" (IEEE Transactions on Antennas and Propagation, Vol. 14, No. 3, pp. 302-307 (1966)), (iv) the Fourier pseudospectral time domain (PSTD) method, which is described, for example, in Sakamoto, S. et al., "Numerical analysis of sound propagation in rooms using the finite difference time domain method”(J.Acoust.Soc.Am., Vol. 120, No. 5, 3008 (2006)) and Sakamoto, S. et al., “Calculation of impulse responses and acoustic parameters in a hall by the finite-difference time-domain method” (Acoustical Science and Technology, Vol. 29, No. 4 (2008)), the contents of which are cited herein.

[0090] The audio prediction module 335 is configured to predict features of the audio output at the ear of the user of the head mounted device 210, such as spectral content and acoustic group delay. The audio prediction module 335 can determine the predicted audio output at the ear based on the features extracted by the feature extraction module 310, the representation of at least a portion of the user's head generated by the reconstruction module 320, and / or the simulation performed by the sound simulation module 330. The predicted audio output at the ear can be the simulated audio output at the ear generated by the sound simulation module 330. Alternatively, the audio prediction module 335 uses a machine learning model to determine the predicted audio output at the ear, which is described below with respect to Figures 9A-9B For example, the audio prediction module 335 can input the features extracted by the feature extraction module 310 into a machine learning model, which is configured to determine the audio output at the ear based on the features.

[0091] The audio prediction module 335 can be trained using one or more machine learning techniques directly based on input data of images and videos of head and ear geometry. In one embodiment, the audio prediction module 335 is periodically retrained at a determined time frequency. The audio prediction module 335 can be trained using feature vectors of a positive training set and a negative training set as input. Different machine learning techniques - such as linear support vector machines (linear SVMs), enhancements to other algorithms (such as AdaBoost), neural networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, boosted stumps, nearest neighbors, k-nearest neighbors, kernel machines, probabilistic models, conditional random fields, Markov random fields, manifold learning, generalized linear models, generalized exponential models, kernel regression, or Bayesian regression - can be used in different embodiments. The following is a description of the various machine learning techniques used in the present invention. Fig. 9A The training audio prediction module 335 is described in more detail.

[0092] The equalizing filter generator 345 generates an equalizing filter customized for the user. In one embodiment, the equalizing filter generator 345 generates an equalizing filter based on the predicted audio output at the user's ear predicted by the audio prediction module 335. In another embodiment, the equalizing filter generator 345 generates an equalizing filter based on the audio output at the user's ear simulated by the sound simulation module 330. As described elsewhere herein, the equalizing filter is configured to adjust one or more acoustic parameters of the audio output for the user when applied to the audio output of the head mounted device 210. For example, the equalizing filter can be configured to adjust other acoustic parameters, such as pitch, dynamics, timbre, etc. The equalizing filter can be a high pass filter, a low pass filter, a parameter personalized equalizing filter, a graphic equalizing filter, or any other suitable type of personalized equalizing filter. In some embodiments, the equalizing filter generator 345 selects an equalizing filter from a set of existing equalizing filters, adjusts the parameters of the existing equalizing filter, generates a new equalizing filter, or adjusts the equalizing filter previously generated by the equalizing filter generator 345 based on the predicted audio output at the user's ear. The equalization filter generator 345 can provide the equalization filter to the head mounted device 210, and the head mounted device 210 can use the equalization filter to provide personalized audio content to the user. In addition or alternatively, the equalization filter generator 345 can provide the equalization filter to the online system 240 to store the equalization filter in association with the user profile of the online system 240.

[0093] Example Method

[0094] Figure 4A is an example view of an imaging device 225 capturing a head image of a user 405 according to one or more embodiments. Figure 4AIn an embodiment of the present invention, the imaging device 225 captures an image including at least the user's ears. The imaging device 225 can also capture images of the user's head at different angles and orientations. For example, the user 405 (or some other party) can position the imaging device 225 at different positions relative to his / her head so that the captured images cover different parts of the user's 405 head. In addition, the user 405 can hold the imaging device 225 at different angles and / or distances relative to the user 405. For example, the user 405 can hold the imaging device 225 at an arm's length directly in front of the user's 405 face and use the imaging device 225 to capture an image of the user's 405 face. The user 405 can also hold the imaging device 225 at a distance shorter than the arm's length, while the imaging device 225 points to the side of the user's 405 head to capture images of the user's 405 ears and / or shoulders. In some embodiments, the imaging device 225 is positioned to capture images of the user's left ear and the user's right ear. Alternatively, the imaging device 225 can capture a 180-degree panoramic view of the user's head so that both ears are captured in a single image or video.

[0095] In some embodiments, the imaging device 225 uses feature recognition software and automatically captures images when features of interest (e.g., ears, shoulders) are identified. Additionally or alternatively, the imaging device 225 may prompt the user to capture an image when the feature of interest is in the field of view of the imaging device 225. In some embodiments, the imaging device 225 includes an application with a graphical user interface (GUI) that guides the user 405 to capture multiple images of the user 405 head from specific angles and / or distances relative to the user 405. For example, the GUI may request a frontal image of the user 405 face, an image of the user 405 right ear, and an image of the user 405 left ear. The imaging device 225 may also determine whether the image is suitable for use by the equalization system 230 (e.g., based on image quality, features captured in the image).

[0096] Figure 4B According to one or more embodiments, Figure 4A 4. A side view of an image of a user 405 captured by the imaging device 225 in FIG. 4. The focus of the captured image is the user's ear 407. In some embodiments, the equalization system 230 may use Figure 4B The images shown are used to determine features associated with the user's ears 407 and / or the user's head. The imaging device 225 may capture additional images to determine additional features associated with the user's head.

[0097] Although Figure 4A The imaging device 225 is shown capturing an image of the head of the user 405 , but the imaging device 225 may also capture an image of a user wearing a head mounted device (eg, head mounted device 100 , head mounted device 105 ).

[0098] Figure 5A is an example view of an imaging device 225 capturing an image of a user 405 wearing a head mounted device 510, according to one or more embodiments. The head mounted device 510 may be an embodiment of the head mounted device 210, some near eye display that includes an audio output (e.g., a speaker), or some other head mounted display that includes an audio output.

[0099] Figure 5B According to one or more embodiments, Figure 5A 2 is a side view of an image of a user 405 wearing a head mounted device 510 captured by an imaging device 225 in FIG. 2. The equalization system 230 can determine features associated with the user's ear 407 with respect to the position of the head mounted device 510, as described in more detail below. In one embodiment, the head mounted device 510 includes one or more transducers, and at least one of the one or more transducers is captured in FIG. Figure 5A In this way, the equalization system 230 can determine the distance between the user's ear 407 and the one or more transducers.

[0100] Visual models benefit from operating on scale and orientation information. There are scenarios where scale or orientation information can be encoded, but these scenarios are generally not trivial. Therefore, in another embodiment, a head-mounted device (e.g., head-mounted device 210) includes one or more visual markers for determining the position of the head-mounted device relative to the user's ears when the user wears the head-mounted device. As described above and elsewhere in this article, a machine learning-based prediction engine uses images and videos of human heads and ears to predict personalized acoustic transfer functions from head-mounted devices measured at the user's ears. Therefore, accurate information about the size and orientation of visually captured anthropometric features is a key requirement for images and videos to be useful for models. Various methods can be designed to provide this information, such as including reference visual objects with known dimensions (e.g., coins) or markers (e.g., multiple points) drawn on features of interest (e.g., ears and eyeglass frames) (their relative distances are measured with a ruler in captured images and videos). However, these methods are cumbersome and / or unreliable, and are therefore not suitable for product applications.

[0101] One approach to eliminating the ambiguity of the size and orientation of anthropometric features is to use markers designed into the head-mounted device for the explicit purpose of providing a visual reference in the data. Thus, in one embodiment, images and / or videos are taken while the user is wearing the head-mounted device with a normal fit. Since the size of these markers is known by design, and their orientation relative to the head and ears is expected to be consistent on each user, the markers can achieve the desired properties of a reliable visual reference within the image. Furthermore, a unique marker design associated with each head-mounted device model with unique dimensions can be used to identify the product model in the image, from which exact information about its industrial design can be inferred.

[0102] Fig. 6A 6 is an example view of an imaging device 225 capturing an image of a user 405 wearing a head mounted device 610 including a plurality of markers 615, in accordance with one or more embodiments. The head mounted device 610 may be an embodiment of the head mounted device 210, some near eye display including an audio output (e.g., a speaker), or some head mounted display including an audio output.

[0103] Figure 6B According to one or more embodiments, Fig. 6A 25 captured by the imaging device 225 in . The head-mounted device 610 captured in the image includes four markings 615a, 615b, 615c, and 615d along its right temple arm 612. The head-mounted device 610 can be symmetrical such that the head-mounted device 610 includes identical markings at corresponding locations on the left temple arm (not shown). In other embodiments, the head-mounted device 610 can include any other suitable number of markings (e.g., one, three, ten) along the right temple arm, the left temple arm, and / or the front of the frame. Figure 6B In an embodiment, each marker 615 has a unique shape and size so that each marker 615 can be easily identified by the equalization system 230. Alternatively, the markers 615 can be substantially the same size and / or shape. In addition, the size of the head mounted device 610 and the position of the markers 615 relative to the head mounted device 610 are known. The equalization system 230 can use Figure 6B The image shown can be used to determine information about the user's ear 407 relative to the head mounted device 610. For example, the equalization system 230 can determine the distance between each marker and a point on the user's ear 407.

[0104] Imaging system 220 may capture one or more images, such as Figure 4B , 5B6B and provide them to the equalization system 230 to generate an equalization filter for the user 405. The equalization system 230 can also receive additional images of other views of the user's ears and / or head from the imaging device 225. The equalization system 230 can determine the audio output at the user's ears based on the images. In addition, the images can be used to train one or more components of the equalization system 230, which is described in more detail below.

[0105] Determining the Equalization Filter Based on Simulation

[0106] A high-fidelity audio experience for a head-mounted device requires that its audio output matches a consistent target response of the user's ears in terms of its spectral content and acoustic time delay. For output modules built into the device frame, a static, non-personalized EQ tuned on a mannequin and / or ear coupler is insufficient to provide such high-fidelity audio output, because the audio heard by each user is affected by multiple sources of variation, such as their anthropometric characteristics (e.g., pinna size and shape), inconsistencies in fittings, sensitivity of transducer components to environmental factors, manufacturing tolerances, etc. Among them, variations between people and between fit levels account for the largest part of audio output variability, and are determined by the shape of the user's head and / or ears and the relative position between the audio output module on the head-mounted device and the user's ears.

[0107] For open-frame audio output head-mounted devices consisting of a frame and speaker modules embedded in the temple arms, industrial design practices can be adopted so that the wearing conditions of the user during normal use of the device are repeatable and stable, thereby minimizing the variation between the fit of the audio output. However, eliminating the variation between people requires knowledge of the audio output at the user's ear, which can be used to compensate for the variation by applying a personalized inverse equalization filter, as described in this article. One way to obtain this knowledge is to place a microphone at the entrance of the ear canal location to measure the raw response of the audio output. The practical application of this method poses challenges in terms of comfort and aesthetics for the industrial design, as well as challenges in terms of ease of use for the user experience. Therefore, an alternative method is needed to measure or predict the audio output at the wearer's ear.

[0108] In one embodiment, a method for achieving this goal includes building a machine learning model that is capable of reconstructing the 3D geometry of a human head and / or ear wearing a head-mounted device based on videos and images, the model being trained using a dataset consisting of images and videos of human subjects wearing head-mounted devices and high-quality 3D scanned meshes of the heads and ears of the corresponding users. The reconstructed 3D geometry is then used as input to a numerical simulation engine for acoustic propagation of the head-mounted device output to the ears, predicting the audio output observed at the user's ears. The predicted response can be used to generate a device-specific personalized equalization filter for the user's audio.

[0109] Figure 7 An example method of generating an equalization filter for a user based on a representation of the user's ears according to one embodiment is shown. These steps may be performed by Figure 2 The steps may be performed by one or more components of the illustrated system 200 (eg, the equalization system 230). In other embodiments, the steps may be performed in an order different from that described herein.

[0110] The equalization system 230 receives 710 one or more images of at least a portion of the user's head. In one embodiment, the equalization system 230 receives one or more images of the user's ears, the user's head, and / or the user wearing the head mounted device 210. For example, the equalization system 230 receives Figure 4B The image shown. The image may be captured using an imaging device 225 associated with a user device (eg, a cell phone).

[0111] The equalization system 230 generates 720 a representation of at least a portion of the user's head based on the one or more images. In some embodiments, the equalization system 230 generates a representation of one or both ears of the user. Alternatively, the equalization system 230 may generate a representation of the user's head that includes one or both ears of the user. The generated representation may be a 3D mesh representing the user's ears and / or head or a PCA-based representation, which is described below with respect to Figures 8A-8B Described in more detail.

[0112] The equalization system 230 performs 730 a simulation of audio propagation from an audio system included in the head mounted device to the user's ear based on the representation of the user's ear. The audio system may be a transducer array coupled to the left temple arm and / or the right temple arm of the head mounted device 210. The equalization system 230 determines a predicted audio output response based on the simulation. For example, the equalization system 230 may determine one or more acoustic parameters (e.g., pitch, frequency, volume, balance, etc.) perceived by the user based on the simulation.

[0113] The equalization system 230 generates 740 an equalization filter based on the predicted audio output response. In this way, the user can experience a customized audio environment provided by the head mounted device 210. For example, due to the anthropometric characteristics of the user, the predicted audio output response may have a higher than average frequency, and the equalization system 230 generates an equalization filter that reduces the frequency of the audio content provided to the user. In some embodiments, the equalization system 230 provides the equalization filter to the head mounted device 210 so that the head mounted device 210 can use the equalization filter to adjust the audio content provided to the user. In addition, the equalization system 230 can provide the equalization filter to the online system 240, and the online system 240 can store the equalization filter to the profile of the user associated with the online system 240 (e.g., a social network profile).

[0114] In some embodiments, the representation of the user's ear described above is generated using a trained model (eg, a PCA model). Employing machine learning techniques allows the reconstruction module 320 to generate a more accurate representation of the user's ear and / or head. Fig. 8A 860 is a block diagram of a trained PCA-model 860 according to one or more embodiments. A machine learning process may be used to generate a PCA-based representation of a user's ear and determine the user's audio output response.

[0115] The reconstruction module 320 receives information (e.g., features from an image of a user's head) from the feature extraction module 310 and / or the depth map generator 315. Based on the information, the reconstruction module 320 generates a PCA-based representation of the user's head using the PCA model 860. In one embodiment, the PCA-based representation also includes a representation of the head-mounted device. In this way, the reconstruction module 320 can use a PCA model 860 that is trained to generate a PCA-based representation, in which a human head or a head shape (e.g., an ear shape) wearing a head-mounted device is represented as a linear combination of the three-dimensional shape of the head or head features of a representative test subject wearing the head-mounted device. In other embodiments, the PCA model 860 is trained to generate a PCA-based representation of a head-mounted device (e.g., the head-mounted device 210), which is represented as a linear combination of the three-dimensional shape of a representative image of the head-mounted device. The PCA model 860 can also be trained to generate a PCA-based representation, in which the shape of a human head or a human head feature (e.g., an ear shape) is represented as a linear combination of the three-dimensional shape of a representative test subject's head or head features. In other embodiments, the PCA model 860 can combine the PCA-based representation of the head with the PCA-based representation of the head mounted device to obtain a PCA-based representation of the head wearing the head mounted device. Alternatively, the PCA model 860 can be trained to produce a PCA-based representation in which the shape of a human head or head feature (e.g., ear shape) wearing a head mounted device (e.g., head mounted device 210) is represented as a linear combination of the three-dimensional shape of the representative test subject's head or head feature when wearing the head mounted device.

[0116] Taking the PCA analysis of the ear shape of a head wearing a head mounted device as an example, the three-dimensional shape of the random ear shape E can be expressed as follows:

[0117] E=∑(α i ×ε i ) (1)

[0118] Among them, α i represents the i-th principal component (i.e., the i-th representative ear shape in three dimensions), ε i Denotes the PCA coefficient of the ith principal component. The number of principal components (the number of "i") is selected to be less than the total number of test subjects for which measured audio output responses were provided. In one example, the number of principal components is 5 to 10.

[0119] In some embodiments, a PCA-based representation is generated using a representation of the head shape of a test subject wearing a head mounted device and its measured audio output response, so that a more accurate equalization filter can be generated by simulation using a PCA-based representation obtained from the PCA model 860 compared to simulating the three-dimensional mesh geometry of the same user's head wearing the head mounted device. The test subject described herein refers to a human or a physical model of a human, whose head shape geometry (or head shape image) and audio output response are already known (i.e., "measured audio output response"). In order to obtain the audio output response, the test subject can be placed in an anechoic chamber and exposed to sound from one or more transducers, wherein the speakers are placed at the test subject's ears. In some embodiments, the audio output response of a test head mounted device (including a test transducer array) worn by the test subject is measured. The test head mounted device is substantially the same as the head mounted device worn by the user.

[0120] like Fig. 8A As shown, PCA model 860 provides a PCA-based representation to sound simulation module 330, and sound simulation module 330 performs a simulated audio output response using the PCA-based representation. Equalization system 230 can compare the measured audio output response of the test object with the simulated audio output response to update PCA model 860, which is described below with respect to Figure 8B 860 is described in more detail. After the PCA model is determined and / or updated, the PCA model 860 is trained using images of the test subject's head wearing the head-mounted device and their PCA-based representations according to the PCA model 860. The trained PCA model 860 can predict or infer the PCA-based representation of the head of the user wearing the head-mounted device from the image of the user's head wearing the head-mounted device. In some embodiments, the trained PCA model 860 can predict or infer the PCA-based representation of the head of the user wearing the head-mounted device from the image of the user's head and other images of the head-mounted device.

[0121] In some embodiments, the generation of the PCA model 860 and the training of the PCA model 860 can be performed offline. The trained PCA model 860 can then be deployed in the reconstruction module 320 of the equalization system 230. Using the trained PCA model 860 enables the reconstruction module 320 to generate PCA-based user representations in a robust and efficient manner.

[0122] Figure 8B is a flow chart of generating and updating PCA model 860 according to one or more embodiments. In one embodiment, Figure 8BThe process is performed by components of the equalization system 230. In other embodiments, other entities may perform some or all steps of the process. Likewise, embodiments may include different steps and / or additional steps, or may perform the steps in a different order.

[0123] Equalization system 230 determines 810 an initial PCA model. In some embodiments, equalization system 230 determines the initial PCA model by selecting a subset of the test subject's head (or a portion thereof) as principal components for representing random head shapes or features of head shapes.

[0124] The equalization system 230 determines 820 a PCA-based representation of the test image using the current PCA model. For example, the initial PCA model processes an image of a test subject's head wearing a test headset that includes a test transducer array to determine a PCA-based representation of the test subject's head or a portion of the test subject's head (e.g., an ear) while the test headset is worn. That is, the head shapes (or shapes of head portions) of all test subjects wearing the headset are represented as a linear combination of a subset of the test subjects' head shapes multiplied by the corresponding PCA coefficients, as explained above with reference to equation (1). Note that the test headset is substantially identical to the headset worn by the user.

[0125] The equalization system 230 performs 830 one or more simulations using the PCA-based representation to generate a simulated audio output response. Figure 3 As described, one or more simulations are performed on the PCA-based representation using one or more of the BEM, FEM, FDTD method, or PSTD method. As a result of the simulation, the equalization system 230 obtains a simulated audio output response of the test object based on the current PCA model.

[0126] Equalization system 230 determines 840 whether the difference between the measured audio output response and the simulated audio output response of the test subject is greater than a threshold. The difference may be the sum of the magnitudes of the differences between the measured audio output response and the simulated audio output response of each test subject.

[0127] If the difference is greater than the threshold, the equalization system 230 updates 850 the PCA model to a new current PCA model. The updating of the PCA model may include increasing or decreasing the number of principal components, updating PCA coefficient values, or updating representative shapes. The process then returns to determining 820 a new set of PCA-based representations based on the updated current PCA model, and repeats subsequent steps.

[0128] If the equalization system 230 determines 840 that the difference is less than or equal to the threshold, then the current PCA model is finally determined 860 as the PCA model for deployment (i.e., for the above description of Figure 7 The described equalization system 230 uses).

[0129] Use the trained model to determine the audio output response

[0130] In another embodiment, the equalization system 230 uses a machine learning model to determine the audio output response. The machine learning model can be trained with a data set consisting of images and videos of human subjects wearing head mounted devices and audio output responses measured at the ears of the corresponding subjects, and the machine learning model will be able to predict the audio output response for a new user based on the images and videos of the new user's head and ear geometry. Therefore, in this embodiment, the machine learning model directly calculates the equalization filter based on the anthropometric features visually extracted from the images and videos.

[0131] Fig. 9A A machine learning process for predicting an audio output response according to an embodiment is shown. The feature extraction module 310 receives an image of a user's head including at least the user's ears. The feature extraction module 310 extracts features describing the user's ears and provides the extracted features to the audio prediction module 335. The audio prediction module 335 uses a response model 970 (i.e., a machine learning model) to predict an audio output response based on the features of the user's ears. The response model 970 is generated and trained using images of additional users, their relevant features, and measured audio response profiles. In some embodiments, the response model 970 can be updated by comparing the predicted audio output response of the additional user with the measured audio output response of the additional user. The additional user described herein refers to a human or a physical model of a human whose anthropometric features and audio output response are known. The anthropometric features can be determined by a person or another model. In order to obtain an audio output response, the additional user can be placed in an anechoic chamber and exposed to sounds from one or more transducers, wherein a microphone is placed at the additional user's ear. In some embodiments, the audio output response of a test head mounted device (including a test transducer array) worn by the additional user is measured. The test head mounted device is substantially the same as the head mounted device worn by the user. The trained response model 970 can be used to predict the audio output response, which is described below with respect to Fig. 9B Described in more detail.

[0132] Fig. 9B A method of generating an equalization filter based on the audio output at the user's ear determined using the response model 970 is shown according to one embodiment. These steps may be performed by Figure 2 The process is performed by one or more components of the system environment 200 shown (e.g., the balancing system 230). In one embodiment, the process uses the above Fig. 9AThe method 900 may be performed by the machine learning response model 970 described herein. The method 900 may include fewer or more steps than described herein.

[0133] The equalization system 230 receives 910 one or more images of the user's ears and / or head. In one embodiment, the equalization system 230 receives one or more images of the user's ears, the user's head, and / or the user wearing the head mounted device 210 (e.g., Figure 4B , 5B 6B). An imaging device 225 (eg, a cell phone) may be used to capture the images.

[0134] The equalization system 230 identifies 920 one or more features from the image depicting the user's ear. The features may describe anthropometric information related to the user's ear and / or head (e.g., size, location, shape). The features may be based on information extracted from the image (e.g., depth information, color information). In some embodiments, the features may be identified relative to the head mounted device. For example, in Figure 6B In an embodiment, the equalization system extracts information related to the position of the head mounted device 510 relative to the user's ear 407 based on the marker 615 to determine the characteristics of the ear.

[0135] The equalization system 230 provides 930 the features as input to a model (e.g., a response model 970). The model is configured to determine an audio output response based on the features. The model is trained using images of the ears of additional users and features extracted from the images, where the audio output response of each additional user is known. The equalization system 230 can periodically retrain the model and use the trained model to predict an audio output response for the user.

[0136] The equalization system 230 generates 940 an equalization filter based on the predicted audio output at the user's ear. The equalization filter is configured to adjust one or more acoustic parameters of the audio content provided by the head mounted device to the user. The equalization system 230 can provide the equalization filter to the head mounted device (e.g., head mounted device 610) so that the head mounted device can use the equalization filter to provide audio content to the user. In addition, the equalization system 230 can provide the equalization filter to the online system 240 for associating the equalization filter with the online profile of the user.

[0137] The trained response model 970 allows the equalization system 230 to quickly and effectively predict the audio output at the user's ear based on the user's image. In this way, the equalization system 230 can generate an equalization filter configured to customize the audio content for the user, enhancing the user's audio experience. In some embodiments, the response model 970 can be used for multiple users and multiple devices. Alternatively, the response model 970 can be customized to a specific device to adjust the audio output of a specific user-device combination. For example, the equalization filter generator 345 can generate a model for the head-mounted device device 100 and another model for the head-mounted device device 105, and the model is generated based on the image of the user wearing the corresponding device and the audio output of the device measured at the user's ear. The equalization system 230 can therefore generate a personalized equalization filter specific to each device of the user.

[0138] In some embodiments, Figure 7 and Figures 8A-8B Aspects of the process shown can be compared with Figures 9A-9B Aspects of the processes shown are combined to enhance the user audio experience. For example, in Fig. 9B In an embodiment of the present invention, the equalization system 230 may additionally generate a 3D representation of the user's ear, such as Figure 7 As described, the 3D representation can be input to the model to generate a predicted audio output response without performing simulations. The equalization system 230 can also generate equalization filters based on a combination of models and / or processes. The equalization system 230 can provide the equalization system 230 to the audio system 215 of the head mounted device 210, which is described in more detail below.

[0139] Audio System

[0140] Fig.10 is a block diagram of an audio system 215 according to one or more embodiments. Figure 1A or Figure 1B The audio system in may be an embodiment of audio system 215. In some embodiments, audio system 215 uses the personalized audio output response generated by equalization system 230 to generate and / or modify audio content for the user. Figure 2 In some embodiments, the audio system 215 includes a transducer array 1010, a sensor array 1020, and an audio controller 1030. Some embodiments of the audio system 215 have components different from those described herein. Similarly, in some cases, functions may be allocated between components in a manner different from that described herein.

[0141] The transducer array 1010 is configured to present audio content. The transducer array 1010 includes a plurality of transducers. A transducer is a device that provides audio content. A transducer may be, for example, a speaker (e.g., speaker 160), a tissue transducer (e.g., tissue transducer 170), some other device that provides audio content, or some combination thereof. A tissue transducer may be configured to be used as a bone conduction transducer or a cartilage conduction transducer. The transducer array 1010 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, the transducer array 1010 may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range, while a moving coil transducer may be used to cover a second part of a frequency range.

[0142] The bone conduction transducer generates sound pressure waves by vibrating bones / tissues in the user's head. The bone conduction transducer can be coupled to a portion of the head mounted device and can be configured to couple to a portion of the user's skull behind the auricle. The bone conduction transducer receives vibration instructions from the audio controller 1030 and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate tissue propagated sound pressure waves that bypass the ear drum and propagate toward the user's cochlea.

[0143] The bone conduction transducer generates sound pressure waves by vibrating one or more portions of the ear cartilage of the user's ear. The bone conduction transducer can be coupled to a portion of the head-mounted device and can be configured to couple to one or more portions of the ear cartilage of the ear. For example, the bone conduction transducer can be coupled to the back of the auricle of the user's ear. The bone conduction transducer can be located anywhere along the ear cartilage around the outer ear (e.g., the auricle, the tragus, some other portion of the ear cartilage, or some combination thereof). Vibrating one or more portions of the ear cartilage can generate: air-borne sound pressure waves outside the ear canal; sound pressure waves generated by tissue, which cause certain portions of the ear canal to vibrate, thereby generating air-borne sound pressure waves in the ear canal; or some combination thereof. The generated air-borne sound pressure waves propagate along the ear canal toward the eardrum.

[0144] The transducer array 1010 generates audio content according to instructions from the audio controller 1030. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that sounds like it originates from a specific direction and / or target area (e.g., objects and / or virtual objects in a local area). For example, the spatialized audio content can make the sound sound like it originates from a virtual singer in a room opposite the user of the audio system 215. The transducer array 1010 can be coupled to a wearable device (e.g., head mounted device 100 or head mounted device 105). In an alternative embodiment, the transducer array 1010 can be a plurality of speakers that are separate from the wearable device (e.g., coupled to an external console).

[0145] In one embodiment, the transducer array 1010 provides audio content to the user using one or more personalized audio output responses generated by the equalization system 230. Each transducer in the transducer array 1010 can use the same personalized audio output response, or each transducer can correspond to a unique personalized audio output response. One or more personalized audio output responses can be received from the equalization system 230 and / or the sound filter module 1080.

[0146] The sensor array 1020 detects sounds in a local area around the sensor array 1020. The sensor array 1020 may include a plurality of acoustic sensors, each of which detects air pressure changes of sound waves and converts the detected sound into an electronic format (analog or digital). The plurality of acoustic sensors may be located on a head-mounted device (e.g., head-mounted device 100 and / or head-mounted device 105), a user (e.g., in the ear canal of the user), a neckband, or some combination thereof. The acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, the sensor array 1020 is configured to monitor the audio content generated by the transducer array 1010 using at least some of the plurality of acoustic sensors. Increasing the number of sensors may improve the accuracy of information (e.g., directionality) describing the sound field generated by the transducer array 1010 and / or the sound from the local area.

[0147] The audio controller 1030 controls the operation of the audio system 215. Fig.10In some embodiments, the audio controller 1030 includes a data store 1035, a DOA estimation module 1040, a transfer function module 1050, a tracking module 1060, a beamforming module 1070, and a sound filter module 1080. In some embodiments, the audio controller 1030 can be located inside the head mounted device. Some embodiments of the audio controller 1030 have different components than those described here. Similarly, the functions can be distributed among the components in a manner different from that described here. For example, some functions of the controller can be performed outside the head mounted device.

[0148] The data storage 1035 stores equalization filters and other data used by the audio system 215. The data in the data storage 1035 may include sounds recorded in a local area of ​​the audio system 215, audio content, head-related transfer functions (HRTFs), transfer functions of one or more sensors, array transfer functions (ATFs) of one or more acoustic sensors, personalized audio output responses, audio profiles, sound source locations, virtual models of the local area, direction of arrival estimates, sound filters, and other related data used by the audio system 215, or any combination thereof.

[0149] The DOA estimation module 1040 is configured to locate sound sources in a local area based in part on information from the sensor array 1020. Localization is the process of determining the location of a sound source relative to the user of the audio system 215. The DOA estimation module 1040 performs a DOA analysis to locate one or more sound sources in a local area. The DOA analysis may include analyzing the intensity, spectrum, and / or arrival time of each sound at the sensor array 1020 to determine the direction from which the sound originated. In some cases, the DOA analysis may include any suitable algorithm for analyzing the ambient acoustic environment in which the audio system 215 is located.

[0150] For example, the DOA analysis may be designed to receive an input signal from the sensor array 1020 and apply a digital signal processing algorithm to the input signal to estimate the direction of arrival. These algorithms may include, for example, a delay and sum algorithm in which the input signal is sampled and the final weighted and delayed versions of the sampled signals are averaged together to determine the DOA. A least mean square (LMS) algorithm may also be implemented to create an adaptive filter. The adaptive filter may then be used, for example, to identify differences in signal strength or differences in arrival time. These differences may then be used to estimate the DOA. In another embodiment, the DOA may be determined by converting the input signal into the frequency domain and selecting a specific bin in the time-frequency (TF) domain to be processed. Each selected TF bin may be processed to determine whether the bin includes a portion of the audio spectrum having a direct path audio signal. Those bins having a portion of the direct path signal may then be analyzed to identify the angle at which the sensor array 1020 receives the direct path audio signal. The determined angle may then be used to identify the DOA of the received input signal. Other algorithms not listed above may also be used alone or in combination with the above algorithms to determine the DOA.

[0151] In some embodiments, the DOA estimation module 1040 can also determine the DOA relative to the absolute position of the audio system 215 within the local area. The position of the sensor array 1020 can be received from an external system (e.g., some other component of the head mounted device, an artificial reality console, a mapping server, a position sensor (e.g., the position sensor 190), etc.). The external system can create a virtual model of the local area, where the local area and the position of the audio system 215 are mapped. The received position information can include the position and / or orientation of part or all of the audio system 215 (e.g., the sensor array 1020). The DOA estimation module 1040 can update the estimated DOA based on the received position information.

[0152] The transfer function module 1050 is configured to generate one or more acoustic transfer functions. In general, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function module 1050 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how a microphone receives sound from a point in space.

[0153] The ATF includes multiple transfer functions that characterize the relationship between a loud sound and a corresponding sound received by an acoustic sensor in the sensor array 1020. Therefore, for a sound source, each acoustic sensor in the sensor array 1020 has a corresponding transfer function. This set of transfer functions is collectively referred to as the ATF. Therefore, for each sound source, there is a corresponding ATF. Note that the sound source can be, for example, someone or something that generates sound in a local area, a user, or one or more transducers of the transducer array 1010. The ATF relative to a specific sound source position of the sensor array 1020 may vary from user to user, because the anatomical structure of a person (e.g., ear shape, shoulders, etc.) affects the sound when the sound travels to the person's ears. Therefore, the ATF of the sensor array 1020 is personalized for each user of the audio system 215.

[0154] In some embodiments, the transfer function module 1050 determines one or more HRTFs for a user of the audio system 215. HRTFs characterize how an ear receives sound from a point in space. The HRTF relative to a person's specific source position is unique to each ear of the person (and unique to the person) because the person's anatomical structure (e.g., ear shape, shoulders, etc.) affects the sound when it travels to the person's ears. In some embodiments, the transfer function module 1050 can determine the HRTF for the user using a calibration process. In some embodiments, the transfer function module 1050 can provide information about the user to the remote system. The remote system uses, for example, machine learning to determine a set of HRTFs customized for the user and provides the customized set of HRTFs to the audio system 215.

[0155] The tracking module 1060 is configured to track the position of one or more sound sources. The tracking module 1060 can compare current DOA estimates and compare them with a stored history of previous DOA estimates. In some embodiments, the audio system 215 can recalculate the DOA estimate periodically, for example once per second, or once per millisecond. The tracking module can compare the current DOA estimate with the previous DOA estimate, and in response to a change in the DOA estimate of the sound source, the tracking module 1060 can determine that the sound source has moved. In some embodiments, the tracking module 1060 can detect a change in position based on visual information received from a head-mounted device or some other external source. The tracking module 1060 can track the movement of one or more sound sources over time. The tracking module 1060 can store values ​​about the number of sound sources and the position of each sound source at each point in time. In response to a change in the value of the number or position of the sound source, the tracking module 1060 can determine that the sound source has moved. The tracking module 1060 can calculate an estimate of the positioning variance. The positioning variance can be used as a confidence level for each determination of the movement change.

[0156] The beamforming module 1070 is configured to process one or more ATFs to selectively emphasize the sound from the sound source in a specific area while weakening the sound from other areas. When analyzing the sound detected by the sensor array 1020, the beamforming module 1070 can combine information from different acoustic sensors to emphasize the relevant sound from the specific area of ​​the local area while weakening the sound from outside the area. The beamforming module 1070 can isolate the audio signal associated with the sound from the specific sound source from other sound sources in the local area based on different DOA estimates from the DOA estimation module 1040 and the tracking module 1060, for example. The beamforming module 1070 can therefore selectively analyze discrete sound sources in the local area. In some embodiments, the beamforming module 1070 can enhance the signal from the sound source. For example, the beamforming module 1070 can apply a sound filter that eliminates signals above a specific frequency, below a specific frequency, or between specific frequencies. Signal enhancement is used to enhance the sound associated with a given identified sound source relative to other sounds detected by the sensor array 1020.

[0157] The sound filter module 1080 determines a sound filter, such as an equalization filter, for the transducer array 1010. In some embodiments, the sound filter causes the audio content to be spatialized so that the audio content sounds as if it originated from a target area. The sound filter module 1080 can generate a sound filter using HRTF and / or acoustic parameters. Acoustic parameters describe the acoustic properties of a local area. Acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the sound filter module 1080 calculates one or more acoustic parameters. In some embodiments, the sound filter module 1080 requests the acoustic parameters from a mapping server (e.g., as described below with reference to Fig.11 In some embodiments, the sound filter module 1080 receives one or more equalization filters, i.e., personalized equalization filters, from the equalization system 230. The sound filter module 1080 provides a sound filter (e.g., a personalized equalization filter) to the transducer array 1010. In some embodiments, the sound filter can cause positive or negative amplification of the sound depending on the frequency.

[0158] Fig.11 1 is a system 1100 including a head mounted device 1105 according to one or more embodiments. In some embodiments, the head mounted device 1105 may be Figure 1A The head mounted device 100 or Figure 1B The head mounted device 105. The system 1100 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Fig.11The system 1100 shown includes a head mounted device 1105, an input / output (I / O) interface 1110 coupled to a console 1115, a network 1120, and a mapping server 1125. Fig.11 An example system 1100 is shown that includes one head mounted device 1105 and one I / O interface 1110, but in other embodiments, any number of these components may be included in the system 1100. For example, there may be multiple head mounted devices, each head mounted device having an associated I / O interface 1110, and each head mounted device and I / O interface 1110 communicating with a console 1115. In alternative configurations, different and / or additional components may be included in the system 1100. Additionally, in some embodiments, the system 1100 may include a plurality of head mounted devices, each head mounted device having an associated I / O interface 1110, and each head mounted device and I / O interface 1110 communicating with a console 1115. Fig.11 The functionality described by one or more components shown in the figure may be combined with Fig.11 The described methods are distributed among the components in different ways. For example, some or all of the functions of the console 1115 can be provided by the head mounted device 1105.

[0159] The head mounted device 1105 includes a display assembly 1130, an optical block 1135, one or more position sensors 1140, and a DCA 1145. Some embodiments of the head mounted device 1105 have a Fig.11 In addition, in other embodiments, by combining Fig.11 The functionality provided by the various components described may be distributed variously among components of the head mounted device 1105 , or captured in a separate component remote from the head mounted device 1105 .

[0160] The display component 1130 displays content to the user based on the data received from the console 1115. The display component 1130 displays content using one or more display elements (e.g., display element 120). The display element can be, for example, an electronic display. In various embodiments, the display component 1130 includes a single display element or multiple display elements (e.g., a display for each eye of the user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof. It should be noted that in some embodiments, the display element 120 can also include some or all of the functionality of the optical block 1135.

[0161] The optical block 1135 can amplify image light received from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or two windows of the head mounted device 1105. In various embodiments, the optical block 1135 includes one or more optical elements. Example optical elements included in the optical block 1135 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 1135 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 1135 may have one or more coatings, such as a partially reflective or anti-reflective coating.

[0162] The magnification and focusing of the image light by the optical block 1135 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content uses almost all of the user's field of view (e.g., approximately 110 degrees diagonally), and in some cases uses all of the field of view to be presented. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0163] In some embodiments, optical block 1135 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberration, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and when optical block 1135 receives image light generated based on the content from the electronic display, optical block 1135 corrects the distortion.

[0164] The position sensor 1140 is an electronic device that generates data indicating the position of the head mounted device 1105. The position sensor 1140 generates one or more measurement signals in response to the movement of the head mounted device 1105. The position sensor 190 is an embodiment of the position sensor 1140. Examples of the position sensor 1140 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, or some combination thereof. The position sensor 1140 may include multiple accelerometers that measure translational movement (forward / backward, up / down, left / right) and multiple gyroscopes that measure rotational movement (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the head mounted device 1105 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine the estimated position of a reference point on the head mounted device 1105. A reference point is a point that can be used to describe the location of the head mounted device 1105. Although a reference point can generally be defined as a point in space, in practice a reference point is defined as a point within the head mounted device 1105.

[0165] The DCA 1145 generates depth information for a portion of a local area. The DCA includes one or more imaging devices and a DCA controller. The DCA 1145 may also include an illuminator. The operation and structure of the DCA 1145 are described above with reference to Figure 1A Described.

[0166] The audio system 1150 provides audio content to the user of the head mounted device 1105. The audio system 1150 is substantially the same as the audio system 215 described above. The audio system 1150 may include one or more acoustic sensors, one or more transducers, and an audio controller. In some embodiments, the audio system 1150 receives one or more equalization filters from the equalization system 230 and applies the equalization filters to one or more transducers. The audio system 1150 may provide spatialized audio content to the user. In some embodiments, the audio system 1150 may request acoustic parameters from the mapping server 1125 via the network 1120. The acoustic parameters describe one or more acoustic characteristics of a local area (e.g., room impulse response, reverberation time, reverberation level, etc.). The audio system 1150 may provide information describing at least a portion of the local area from, for example, the DCA 1145 and / or receive location information of the head mounted device 1105 from the location sensor 1140. The audio system 1150 may generate one or more sound filters using the one or more acoustic parameters received from the mapping server 1125 and provide audio content to the user using the sound filters.

[0167] The I / O interface 1110 is a device that allows a user to send an action request and receive a response from the console 1115. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end the capture of image or video data, or an instruction to perform a specific action within an application. The I / O interface 1110 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 1115. The action request received by the I / O interface 1110 is transmitted to the console 1115, and the console 1115 performs the action corresponding to the action request. In some embodiments, the I / O interface 1110 includes an IMU that captures calibration data indicating an estimated position of the I / O interface 1110 relative to the initial position of the I / O interface 1110. In some embodiments, the I / O interface 1110 may provide tactile feedback to the user according to the instruction received from the console 1115. For example, tactile feedback is provided when an action request is received, or the console 1115 transmits instructions to the I / O interface 1110 , causing the I / O interface 1110 to generate tactile feedback when the console 1115 performs an action.

[0168] The console 1115 provides content to the head mounted device 1105 for processing based on information received from one or more of: the DCA 1145, the head mounted device 1105, and the I / O interface 1110. Fig.11 In the example shown, the console 1115 includes an application store 1155, a tracking module 1160, and an engine 1165. Some embodiments of the console 1115 have a Fig.11 Similarly, the functions described further below may be implemented in different ways in combination with Fig.11 The described approach is distributed among the components of the console 1115. In some embodiments, the functionality discussed herein with reference to the console 1115 can be implemented in the head mounted device 1105 or in a remote system.

[0169] The application storage 1155 stores one or more applications executed by the console 1115. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be responsive to input received from the user via movement of the head mounted device 1105 or the I / O interface 1110. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0170] The tracking module 1160 tracks the movement of the head mounted device 1105 or the I / O interface 1110 using information from the DCA 1145, one or more position sensors 1140, or some combination thereof. For example, the tracking module 1160 determines the location of a reference point of the head mounted device 1105 in a mapping of the local area based on the information from the head mounted device 1105. The tracking module 1160 may also determine the location of an object or virtual object. Additionally, in some embodiments, the tracking module 1160 may use the portion of the data from the position sensor 1140 indicating the location of the head mounted device 1105 and the representation of the local area from the DCA 1145 to predict the future location of the head mounted device 1105. The tracking module 1160 provides the estimated or predicted future location of the head mounted device 1105 or the I / O interface 1110 to the engine 1165.

[0171] The engine 1165 executes the application and receives the position information, acceleration information, velocity information, predicted future position, or some combination thereof of the head mounted device 1105 from the tracking module 1160. Based on the received information, the engine 1165 determines the content to be provided to the head mounted device 1105 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 1165 generates content for the head mounted device 1105 that reflects (mirror) the user's movement in a virtual local area or in a local area that enhances the local area with additional content. In addition, the engine 1165 performs an action within an application executed on the console 1115 in response to an action request received from the I / O interface 1110 and provides feedback to the user that the action has been performed. The feedback provided may be visual or auditory feedback via the head mounted device 1105 or tactile feedback via the I / O interface 1110.

[0172] The network 1120 couples the head mounted device 1105 and / or the console 1115 to the mapping server 1125. The network 1120 may include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, the network 1120 may include the Internet and a mobile phone network. In one embodiment, the network 1120 uses standard communication technologies and / or protocols. Therefore, the network 1120 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 1120 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged through the network 1120 may be represented using techniques and / or formats including image data in binary form (e.g., portable network graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or part of the link may be encrypted using conventional encryption techniques, such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.

[0173] The mapping server 1125 may include a database storing a virtual model describing a plurality of spaces, wherein a location in the virtual model corresponds to a current configuration of a local area of ​​the head mounted device 1105. The mapping server 1125 receives information describing at least a portion of the local area and / or location information of the local area from the head mounted device 1105 via the network 1120. The mapping server 1125 determines a location in the virtual model associated with the local area of ​​the head mounted device 1105 based on the received information and / or location information. The mapping server 1125 determines (e.g., retrieves) one or more acoustic parameters associated with the local area based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The mapping server 1125 may transmit the location of the local area and any acoustic parameter values ​​associated with the local area to the head mounted device 1105.

[0174] Additional configuration information

[0175] The foregoing description of embodiments of the present disclosure is presented for purposes of illustration; it is not intended to be exhaustive or to limit patent rights to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0176] Some parts of this description describe embodiments in terms of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively communicate the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0177] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0178] Embodiments may also relate to apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0179] Embodiments may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0180] Finally, the language used in the specification is selected primarily for readability and instructional purposes and may not be selected to describe or limit the patent rights. Therefore, the scope of the patent rights is not limited by this detailed description, but by any claims of an application filed based thereon. Therefore, the disclosure of the embodiments is intended to illustrate, not to limit, the scope of the patent rights set forth in the appended claims.

Claims

1. A method for generating a personalized equalization filter for a user of an artificial reality system, comprising: receiving an image including an ear of a user; generating a three-dimensional 3D representation of the user's ear based in part on the received image; performing a simulation of sound propagation from an audio source to an ear of a user based on the 3D representation, the simulation generating a simulated frequency response of audio output from the audio source at the ear of the user; and An equalization filter is generated for a user based on a ratio between a simulated frequency response of the audio output from the audio source at the user's ear and a target frequency response, the equalization filter being configured to adjust one or more acoustic parameters of audio content provided to the user.

2. The method according to claim 1, wherein: The user wears a head mounted device in the image of the user's ear, and wherein generating a 3D representation of the user's ear also includes generating a 3D representation including the head mounted device.

3. A method according to claim 1 or claim 2, wherein the equalization filter, when applied to audio content provided to a user, adjusts one or more acoustic parameters of the audio content for the user based on the simulation of sound propagation from the audio source to the user's ears.

4. A method according to claim 3, wherein the audio source is a speaker array of a head-mounted device; and / or wherein the simulation of sound propagation from the audio source to the user's ears simulates audio output at the user's ears.

5. The method according to any one of claims 1 to 2 and 4, wherein generating the 3D representation comprises processing the image of the user's ear using a machine learning model to obtain the 3D representation.

6. The method of claim 5, wherein the 3D representation is generated using a principal component analysis (PCA) model that describes the user's ear as a combination of representative three-dimensional shapes of the test subject's ears.

7. The method according to claim 6, wherein the PCA model is generated by: receiving an image of the test subject's ear and a measured audio output response of the test subject; determining a PCA-based 3D representation of the test subject's ear image based on the initial PCA model; simulating the PCA-based 3D representation to determine a simulated audio output response of the test object; determining a difference between the simulated audio output response and the measured audio output response; generating an updated PCA model by modifying the initial PCA model or by updating one or more intermediate PCA models derived from the initial PCA model based on the determined differences; and When the determined difference is lower than a threshold, the updated PCA model is determined as the PCA model for representing the user's ear.

8. The method according to claim 7, further comprising: According to the PCA model, the machine learning model is trained using the test subject's ear image and the PCA-based 3D representation of the test subject's ear image.

9. The method according to any one of claims 1 to 2, 4 and 6 to 8, wherein The image of the user's ear is captured by a depth camera component of the head-mounted device.

10. A method according to any one of claims 1 to 2, 4 and 6 to 8, wherein the generated 3D representation comprises a three-dimensional mesh of the user's ear.

11. The method according to any one of claims 1 to 2, 4 and 6 to 8, further comprising: The generated equalization filter is provided to a head mounted device, which is configured to employ the equalization filter to provide audio content to a user.

12. The method according to any one of claims 1 to 2, 4 and 6 to 8, further comprising: The generated equalization filter is provided to an online system, wherein the online system stores the generated equalization filter in association with the user's online profile.

13. A non-transitory computer-readable storage medium storing instructions thereon which, when executed by a processor, cause the processor to perform steps comprising: receiving an image including an ear of a user; generating a 3D representation of the user's ear based in part on the received image; performing a simulation of sound propagation from an audio source to an ear of a user based on the 3D representation, the simulation generating a simulated frequency response of audio output from the audio source at the ear of the user; and An equalization filter is generated for a user based on a ratio between a simulated frequency response of the audio output from the audio source at the user's ear and a target frequency response, the equalization filter adjusting one or more acoustic parameters of audio content provided to the user.

14. The non-transitory computer-readable storage medium of claim 13, wherein: The user wears a head mounted device in the image of the user's ear, and wherein generating a 3D representation of the user's ear also includes generating a 3D representation of the head mounted device.

15. The non-transitory computer-readable storage medium of claim 13 or claim 14, wherein the equalization filter, when applied to audio content provided to a user, adjusts one or more acoustic parameters of the audio content for the user based on the simulation of sound propagation from the audio source to the user's ears.

16. The non-transitory computer-readable storage medium of claim 15, wherein the audio source is a speaker array of a head mounted device.

17. The non-transitory computer-readable storage medium of any one of claims 13, 14, and 16, wherein the simulation is based on one of a finite element method (FEM), a boundary element method (BEM), and a finite difference time domain (FDTD) method, and wherein the simulation of sound propagation from the audio source to the user's ear simulates an audio output at the user's ear.

18. The non-transitory computer-readable storage medium according to any one of claims 13, 14 and 16, the steps further comprising: providing the generated equalization filter to a head mounted device, the head mounted device being configured to provide audio content to a user using the equalization filter; And / or, the steps further include: The generated equalization filter is provided to an online system configured to append the generated equalization filter to an online profile of the user.

Images