Audio systems for artificial reality environments

Abstract

An audio system on a head mounted device presents audio content simulating a target artificial reality environment to a user. The system receives audio content from the environment and analyzes the audio content to determine a set of acoustic characteristics associated with the environment. The audio content may be user generated sounds or ambient sounds. After receiving a set of target acoustic characteristics of the target environment, the system determines a transfer function by comparing the set of acoustic characteristics with the acoustic characteristics of the target environment. The system adjusts the audio content based on the transfer function and presents the adjusted audio content to the user. The presented adjusted audio content includes one or more of the target acoustic characteristics of the target environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 450,678, filed on June 24, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0003] background

[0004] The present disclosure relates generally to audio systems, and in particular to audio systems that render sounds of a target artificial reality environment.

[0005] A head-mounted display (HMD) can be used to present virtual and / or enhanced information to a user. For example, an augmented reality (AR) head-mounted device or a virtual reality (VR) head-mounted device can be used to simulate augmented / virtual reality. Traditionally, users of AR / VR head-mounted devices wear headphones to receive or otherwise experience computer-generated sounds. The environment in which the user wears the AR / VR head-mounted device usually does not match the virtual space simulated by the AR / VR head-mounted device, thereby presenting an auditory conflict to the user. For example, musicians and actors usually need to complete rehearsals in a performance space because their playing style and the sound received in the audience area depend on the acoustic effects of the hall. In addition, in games or applications involving user-generated sounds (such as speech, applause, etc.), the acoustic characteristics of the real space where the player is located do not match the acoustic characteristics of the virtual space.

[0006] Overview

[0007] A method for rendering sounds in a target artificial reality environment is disclosed. The method analyzes a set of acoustic characteristics associated with the environment by a controller. The environment can be a room where a user is located. One or more sensors receive audio content (including user-generated sounds and environmental sounds) from the environment. For example, a user can speak, play an instrument or sing in the environment, and the environmental sounds can include fans running and dogs barking, etc. In response to receiving a selection of a target artificial reality environment such as a stadium, a concert hall or a venue, the controller compares the acoustic characteristics of the room where the user is currently located with a set of target acoustic characteristics associated with the target environment. The controller then determines a transfer function, which the controller uses to adjust the received audio content. Therefore, one or more speakers present the adjusted audio content to the user, so that the adjusted audio content includes one or more of the target acoustic characteristics of the target environment. The user perceives the adjusted audio content as if they were in the target environment.

[0008] In some embodiments, the method is performed by an audio system that is part of a head mounted device (e.g., a near eye display (NED), a head mounted display (HMD)). The audio system includes one or more sensors for detecting audio content, one or more speakers for presenting adjusted audio content, and a controller for analyzing acoustic characteristics of an environment with acoustic characteristics of a target environment and determining a transfer function that characterizes a comparison of the two sets of acoustic characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of a head mounted device according to one or more embodiments.

[0011] Figure 2A A sound field is shown in accordance with one or more embodiments.

[0012] Figure 2B A sound field is shown after rendering audio content of a target environment in accordance with one or more embodiments.

[0013] Figure 3 is a block diagram of an example audio system in accordance with one or more embodiments.

[0014] Figure 4 is a process for rendering audio content for a target environment according to one or more embodiments.

[0015] Figure 5 is a block diagram of an example artificial reality system in accordance with one or more embodiments.

[0016] The accompanying drawings depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0017] Detailed Description

[0018] An audio system renders audio content of a target artificial reality environment. When wearing an artificial reality (AR) or virtual reality (VR) device such as a head-mounted device, a user can generate audio content (e.g., speech, music from an instrument, applause, or other noise). The acoustic characteristics of the user's current environment (such as a room) may not match the acoustic characteristics of the virtual space simulated by the AR / VR head-mounted device (i.e., the target artificial reality environment). The audio system renders the audio content generated by the user as if it were generated in the target environment, while also taking into account the ambient sounds in the user's current environment. For example, a user can use a head-mounted device to simulate a vocal performance in a concert hall (i.e., the target environment). When the user sings, the audio system adjusts the audio content (i.e., the sound of the user singing) so that it sounds like the user is singing in a concert hall. Ambient noise in the user's surrounding environment (e.g., dripping water, people talking, or fans running) may be attenuated because the target environment is unlikely to be characterized by these sounds. The audio system takes into account ambient sounds and user-generated sounds that do not have the characteristics of the target environment, and renders the audio content so that it sounds like it is produced in the target artificial reality environment.

[0019] The audio system includes one or more sensors for receiving audio content (including user-generated sounds and ambient sounds around the user). In some embodiments, the audio content can be generated by more than one user in the environment. The audio system analyzes a set of acoustic characteristics of the user's current environment. The audio system receives the user's selection of a target environment. After comparing the original response associated with the acoustic characteristics of the current environment and the target response associated with the acoustic characteristics of the target environment, the audio system determines a transfer function. The audio system adjusts the detected audio content according to the determined transfer function and presents the adjusted audio content to the user through one or more speakers.

[0020] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0021] System Overview

[0022] Figure 1 1 is a diagram of a head mounted device 100 according to one or more embodiments. The head mounted device 100 presents media to a user. The head mounted device 100 includes an audio system, a display 105, and a frame 110. Typically, the head mounted device can be worn on the face of a user so that content is presented using the head mounted device. The content may include audio and visual media content presented through the audio system and the display 105, respectively. In some embodiments, the head mounted device may present audio content to the user only through the head mounted device. The frame 110 enables the head mounted device 100 to be worn on the face of a user and houses components of the audio system. In one embodiment, the head mounted device 100 may be a head mounted display (HMD). In another embodiment, the head mounted device 100 may be a near eye display (NED).

[0023] Display 105 presents visual content to the user of head mounted device 100. The visual content may be part of a virtual reality environment. In some embodiments, display 105 may be an electronic display element, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a quantum organic light emitting diode (QOLED) display, a transparent organic light emitting diode (TOLED) display, some other display, or some combination thereof. Display 105 may be backlit. In some embodiments, display 105 may include one or more lenses that enhance what the user sees when wearing head mounted device 100.

[0024] The audio system presents audio content to the user of the head mounted device 100. The audio system includes, among other components, one or more sensors 140A, 140B, one or more speakers 120A, 120B, 120C, and a controller. The audio system can provide adjusted audio content to the user, rendering the detected audio content as if it were produced in the target environment. For example, the user of the head mounted device 100 may want to practice playing an instrument in a concert hall. The head mounted device 100 will present visual content that simulates the target environment (i.e., the concert hall), as well as audio content that simulates how the user will perceive the sounds in the target environment. Additional details about the audio system are described below in connection with Figure 2A-Figure 5 Discuss.

[0025] Speakers 120A, 120B, and 120C generate sound pressure waves to present to the user according to instructions from controller 170. Speakers 120A, 120B, and 120C can be configured to present adjusted audio content to the user, wherein the adjusted audio content includes at least some acoustic characteristics of the target environment. One or more speakers can generate sound pressure waves by air conduction, which transmits air-borne sound to the user's ears. In some embodiments, the speaker can present content by tissue conduction, wherein the speaker can be a transducer that directly vibrates tissue (e.g., bone, skin, cartilage, etc.) to generate sound pressure waves. For example, speakers 120B and 120C can be coupled to tissue near and / or at the ear and vibrate the tissue to generate sound pressure waves carried by the tissue detected as sound by the cochlea of ​​the user's ear. Speakers 120A, 120B, and 120C can cover different parts of the frequency range. For example, a piezoelectric transducer can be used to cover a first part of the frequency range, and a moving coil transducer can be used to cover a second part of the frequency range.

[0026] Sensors 140A, 140B monitor and capture data about audio content from the user's current environment. The audio content may include user-generated sounds (including users speaking, playing instruments, and singing), as well as environmental sounds (e.g., dogs panting, air conditioners running, and water flowing). Sensors 140A, 140B may include, for example, microphones, accelerometers, other acoustic sensors, or some combination thereof.

[0027] In some embodiments, speakers 120A, 120B, and 120C and sensors 140A and 140B may be located within and / or on frame 110. Figure 1 The head mounted device may include a number and / or type of different Figure 1 The number and / or type of speakers and / or sensors shown.

[0028] The controller 170 instructs the speaker to present the audio content and determines the transfer function between the user's current environment and the target environment. The environment is associated with a set of acoustic characteristics. The acoustic characteristics characterize how the environment responds to the acoustic content (e.g., the propagation and reflection of sound in the environment). The acoustic characteristics can be the reverberation time from the sound source to the head-mounted device 100 for multiple frequency bands, the reverberation level of each frequency band, the direct to reverberant ratio of each frequency band, the early reflection time of the sound from the sound source to the head-mounted device 100, other acoustic characteristics, or some combination thereof. For example, the acoustic characteristics may include the reflection of the signal from the surface in the room and the attenuation of the signal as it propagates in the air.

[0029] The user can use the head mounted device 100 to simulate a target artificial reality environment (i.e., a "target environment"). A user located in a current environment (e.g., a room) can choose to simulate a target environment. The user can select a target environment from a plurality of possible target environment options. For example, the user can select a stadium from a selection list including an opera hall, an indoor basketball court, a music recording studio, etc. The target environment has its own set of acoustic characteristics (i.e., a set of target acoustic characteristics) that characterize how the sound is perceived in the target environment. The controller 170 determines the "original response," i.e., the room impulse response of the user's current environment, based on a set of acoustic characteristics of the current environment. The original response characterizes how the user perceives the sound in his current environment (i.e., the room) at the first position. In some embodiments, the controller 170 can determine the original response at the user's second position. For example, the sound perceived by the user in the center of the room will be different from the sound perceived at the entrance of the room. Therefore, the original response at the first position (e.g., the center of the room) will be different from the original response at the second position (e.g., the entrance of the room). The controller 170 also determines the "target response" based on the target acoustic characteristics, which characterizes how the sound will be perceived in the target environment. Comparing the original response and the target response, the controller 170 determines the transfer function it uses when adjusting the audio content. When comparing the original response and the target response, the controller 170 determines the difference between the acoustic parameters in the user's current environment and the acoustic parameters in the target environment. In some cases, the difference can be negative, in which case the controller 170 eliminates and / or blocks the sound in the user's current environment to obtain the sound in the target environment. In other cases, the difference can be additive, where the controller 170 adds and / or enhances certain sounds to portray the sound in the target environment. The controller 170 can use sound filters to change the sound in the current environment to obtain the sound in the target environment, which is described below with respect to Figure 3 Detailed description is given below. The controller 170 can measure the difference between the sound in the current environment and the sound in the target environment by determining the difference in environmental parameters that affect the sound in the environment. For example, in addition to comparing acoustic parameters such as reverberation and attenuation, the controller 170 can also compare the temperature and relative humidity of the environment. In some embodiments, the transfer function is specific to the user's position in the environment, such as the first position or the second position. The adjusted audio content reflects at least some of the target acoustic characteristics so that the user perceives the sound as if it were produced in the target environment.

[0030] Rendering sounds for the target environment

[0031] Figure 2AA sound field according to one or more embodiments is shown. A user 210 is located in an environment 200 (e.g., a living room). The environment 200 has a sound field 205, which includes ambient noise and user-generated sounds. The sources of ambient noise may include, for example, traffic on a nearby street, a neighbor's dog barking, and other people typing on keyboards in adjacent rooms. The user 210 can generate sounds such as singing, playing guitar, stomping, and talking. In some embodiments, the environment 200 may include multiple users generating sounds. Before wearing an artificial reality (AR) and / or virtual reality (VR) head-mounted device (e.g., head-mounted device 100), the user 210 can perceive sounds based on a set of acoustic characteristics of the environment 200. For example, in a living room that may be filled with many objects, the user 210 can perceive minimal echo when they speak.

[0032] Figure 2B 2 shows a sound field after rendering the audio content of the target environment according to one or more embodiments. User 210 is still located in environment 200 and is wearing head mounted device 215. Head mounted device 215 is Figure 1 An embodiment of the head mounted device 100 described in the figure renders the audio content so that the user 210 perceives the adjusted sound field.

[0033] The head mounted device 215 detects the audio content in the environment of the user 210 and presents the adjusted audio content to the user 210. Figure 1 , the head mounted device 215 includes an audio system having at least one or more sensors (e.g., sensors 140A, 140B), one or more speakers (e.g., speakers 120A, 120B, 120C), and a controller (e.g., controller 170). The audio content in the environment 200 of the user 210 may be generated by the user 210, other users in the environment 200, and / or ambient sounds.

[0034] The controller identifies and analyzes a set of acoustic characteristics associated with the environment 200 by estimating a room impulse response that characterizes the user 210's perception of sounds emitted within the environment 200. The room impulse response is associated with the user 210's perception of sounds at a particular location in the environment 200, and if the user 210 changes his position within the environment 200, the room impulse response will change. The room impulse response can be generated by the user 210 before the head-mounted device 215 renders content for AR / VR simulation. The user 210 can generate a test signal using, for example, a mobile device, in response to which the controller measures the impulse response. Alternatively, the user 210 can generate an impulse noise, such as clapping, to generate an impulse signal measured by the controller. In another embodiment, the head-mounted device 215 may include an image sensor (e.g., a camera) to record image and depth data associated with the environment 200. The controller can use sensor data and machine learning to simulate the size, layout, and parameters of the environment 200. Thus, the controller can learn the acoustic characteristics of the environment 200 to obtain the impulse response. The controller uses the room impulse response to define a raw response that characterizes the acoustic characteristics of the environment 200 before the audio content is adjusted. Estimating the acoustic characteristics of the room is described in more detail in U.S. Patent Application No. 16 / 180,165, filed on November 5, 2018, which is incorporated herein by reference in its entirety.

[0035] In another embodiment, the controller may provide visual information detected by the head mounted device 215 to a mapping server, wherein the visual information describes at least a portion of the environment 200. The mapping server may include a database of environments and their associated acoustic characteristics, and may determine a set of acoustic characteristics associated with the environment 200 based on the received visual information. In another embodiment, the controller may query the mapping server for location information, in response to which the mapping server may retrieve the acoustic characteristics of the environment associated with the location information. Figure 5 The use of a mapping server in the context of an artificial reality system is discussed in more detail.

[0036] The user 210 may specify a target artificial reality environment for rendering the sound. For example, the user 210 may select the target environment through an application on a mobile device. In another embodiment, the head mounted device 215 may be pre-programmed to render a set of target environments. In another embodiment, the head mounted device 215 may be connected to a mapping server that includes a database listing available target environments and associated target acoustic properties. The database may include real-time simulations of target environments, data on impulse responses measured in target environments, or algorithmic reverberation methods.

[0037] The controller of the head mounted device 215 uses the acoustic characteristics of the target environment to determine the target response, and then compares the target response and the original response to determine the transfer function. The original response characterizes the acoustic characteristics of the user's current environment, while the target response characterizes the acoustic characteristics of the target environment. The acoustic characteristics include reflections from different directions in the environment (with specific timing and amplitude). The controller uses the difference between the reflections in the current environment and the reflections in the target environment to generate a difference reflection pattern (pattern), characterized by a transfer function. Based on the transfer function, the controller can determine the head-related transfer function (HRTF) required to convert the sound generated in the environment 200 into the sound perceived in the target environment. The HRTF characterizes how the user's ears receive sound from a point in space and varies according to the user's current head position. The controller applies the HRTF corresponding to the reflection direction at the timing and amplitude of the reflection to generate the corresponding target reflection. The controller repeats this process in real time for all difference reflections so that the user perceives the sound as if it were generated in the target environment. HRTF is described in detail in U.S. patent application Ser. No. 16 / 390,918, filed on April 22, 2019, which is incorporated herein by reference in its entirety.

[0038] After wearing the head mounted device 215, the user 210 may generate some audio content detected by the sensors on the head mounted device 215. For example, the user 210 may stomp on the ground physically located in the environment 200. The user 210 selects a target environment, such as a Figure 2B 200, for example, a target response is determined by the controller. The controller determines a transfer function for a specified target environment. The controller of the head mounted device 215 convolves the transfer function with sounds generated within the environment 200 (e.g., the stomping of the user 210's feet) in real time. The convolution adjusts the acoustic characteristics of the audio content based on the target acoustic characteristics, thereby producing adjusted audio content. The speakers of the head mounted device 215 present the adjusted audio content to the user, which now includes one or more of the target acoustic characteristics. Ambient sounds in the environment 200 that do not have features in the target environment are suppressed so that the user 210 does not perceive them. For example, the barking of a dog in the sound field 205 does not appear in the adjusted audio content presented via the adjusted sound field. The user 210 will perceive the sound of their stomping feet as if they were in the target environment of the indoor tennis court, which may not include the barking of a dog.

[0039] Figure 33 is a block diagram of an example audio system according to one or more embodiments. Audio system 300 may be a component of a head mounted device (e.g., head mounted device 100) that provides audio content to a user. Audio system 300 includes a sensor array 310, a speaker array 320, and a controller 330 (e.g., controller 170). Figure 1-2B The audio system described in is an embodiment of the audio system 300. Some embodiments of the audio system 300 include other components in addition to those described herein. Similarly, the functional distribution of the components can be different from that described herein. For example, in one embodiment, the controller 330 can be external to the head mounted device, rather than embedded in the head mounted device.

[0040] The sensor array 310 detects audio content from within the environment. The sensor array 310 includes a plurality of sensors, such as sensors 140A and 140B. The sensor may be an acoustic sensor (e.g., a microphone, a vibration sensor, an accelerometer, or any combination thereof) configured to detect sound pressure waves. The sensor array 310 is configured to monitor the sound field within the environment (e.g., the sound field 205 in the room 200). In one embodiment, the sensor array 310 converts the detected sound pressure waves into an electrical format (analog or digital), which the sensor array then sends to the controller 330. The sensor array 310 detects user-generated sounds (e.g., a user talking, singing, or playing an instrument), as well as environmental sounds (e.g., a fan running, dripping water, or a dog barking). The sensor array 310 distinguishes between user-generated sounds and environmental noise by tracking the sound source, and stores the audio content in the data storage 340 of the controller 330 accordingly. The sensor array 310 may perform position tracking of the source of the audio content within the environment by direction of arrival (DOA) analysis, video tracking, computer vision, or any combination thereof. The sensor array 310 may use beamforming technology to detect audio content. In some embodiments, the sensor array 310 includes sensors that are different from those used to detect acoustic pressure waves. For example, the sensor array 310 may include an image sensor, an inertial measurement unit (IMU), a gyroscope, a position sensor, or a combination thereof. The image sensor may be a camera configured to perform video tracking and / or communicate with the controller 330 for computer vision. Beamforming and DOA analysis are described in more detail in U.S. Patent Application No. 16 / 379,450 filed on April 9, 2019 and U.S. Patent Application No. 16 / 016,156 filed on June 22, 2018, both of which are incorporated herein by reference in their entirety.

[0041] The speaker array 320 presents audio content to the user. The speaker array 320 includes a plurality of speakers, such as Figure 1The speakers 120A, 120B, 120C in the speaker array 320. The speakers in the speaker array 320 are transducers that transmit sound pressure waves to the ears of the user wearing the head-mounted device. The transducer can transmit audio content through air conduction, where the sound pressure waves propagated by the air reach the cochlea of ​​the user's ear and are perceived as sound by the user. The transducer can also transmit audio content through tissue conduction (such as bone conduction, cartilage conduction, or some combination thereof). The speakers in the speaker array 320 can be configured to provide sound to the user over the entire frequency range. For example, the entire frequency range is 20Hz to 20kHz, which is generally around the average range of human hearing. The speakers are configured to transmit audio content over various frequency ranges. In one embodiment, each speaker in the speaker array 320 operates over the entire frequency range. In another embodiment, one or more speakers operate in a low sub-range (e.g., 20Hz to 500Hz), while a second group of speakers operate in a high sub-range (e.g., 500Hz to 20kHz). The sub-range of the speaker may partially overlap with one or more other sub-ranges.

[0042] The controller 330 controls the operation of the audio system 300. The controller 330 is substantially similar to the controller 170. In some embodiments, the controller 330 is configured to adjust the audio content detected by the sensor array 310 and instruct the speaker array 320 to present the adjusted audio content. The controller 330 includes a data storage 340, a response module 350, and a sound adjustment module 370. The controller 330 can query the mapping server for the acoustic characteristics of the user's current environment and / or the acoustic characteristics of the target environment, which will be related to the acoustic characteristics of the user's current environment and / or the acoustic characteristics of the target environment. Figure 5 Further description. In some embodiments, controller 330 can be located inside the head mounted device. Some embodiments of controller 330 have components different from those described herein. Similarly, functions can be distributed between components in a manner different from that described herein. For example, some functions of controller 330 can be performed outside the head mounted device.

[0043] The data storage 340 stores data for use by the audio system 300. The data in the data storage 340 may include a plurality of target environments that may be selected by the user, a set of acoustic characteristics associated with a target environment, a target environment selected by the user, impulse responses measured in the user's current environment, head-related transfer functions (HRTFs), sound filters, and other data related to the use of the audio system 300, or any combination thereof.

[0044] The response module 350 determines an impulse response and a transfer function based on the acoustic characteristics of the environment. The response module 350 determines an original response that characterizes the acoustic characteristics of the user's current environment (e.g., environment 200) by estimating an impulse response to an impulse sound. For example, the response module 350 can use an impulse response to a single drum sound in the room where the user is located to determine the acoustic parameters of the room. The impulse response is associated with a first position of the sound source, which can be determined by the sensor array 310 as described above through DOA and beamforming analysis. The impulse response may change as the sound source and the location of the sound source change. For example, the acoustic characteristics of the room where the user is located may be different in the center and in the periphery. The response module 350 accesses a list of target environment options and their target responses from the data storage 340, which characterizes their associated acoustic characteristics. Subsequently, the response module 350 determines a transfer function that characterizes the target response compared to the original response. The original response, the target response, and the transfer function are all stored in the data storage 340. The transfer function may be unique to a specific sound source, the location of the sound source, the user, and the target environment.

[0045] The sound adjustment module 370 adjusts the sound according to the transfer function and instructs the speaker array 320 to play the adjusted sound accordingly. The sound adjustment module 370 convolves the transfer function of the specific target environment stored in the data storage 340 with the audio content detected by the sensor array 310. The convolution causes the detected audio content to be adjusted based on the acoustic characteristics of the target environment, wherein the adjusted audio content has at least some target acoustic characteristics. The convolved audio content is stored in the data storage 340. In some embodiments, the sound adjustment module 370 generates a sound filter based in part on the convolved audio content, and then instructs the speaker array 320 to present the adjusted audio content accordingly. In some embodiments, the sound adjustment module 370 considers the target environment when generating the sound filter. For example, in a target environment where all other sound sources except the sound generated by the user are quiet, such as a classroom, the sound filter can attenuate the ambient sound pressure wave while amplifying the sound generated by the user. In a noisy target environment (such as a busy street), the sound filter can amplify and / or enhance the sound pressure wave that matches the acoustic characteristics of the busy street. In other embodiments, the sound filter can target a specific frequency range through a low pass filter, a high pass filter, and a band pass filter. Alternatively, the sound filter can enhance the detected audio content to reflect the audio content in the target environment. The generated sound filter is stored in the data storage 340.

[0046] Figure 4 4 is a process 400 of rendering audio content of a target environment according to one or more embodiments. An audio system, such as audio system 300, performs the process. Figure 4 The process 400 may be performed by a component of the device (eg Figure 3 In other embodiments, other entities (e.g., Figure 1 Components of the head mounted device 100 and / or Figure 5 The components shown in the figure) can perform some or all steps of the process. Likewise, embodiments can include different and / or additional steps, or perform the steps in a different order.

[0047] The audio system analyzes 410 a set of acoustic characteristics of the environment (e.g., the room in which the user is located). Figure 1-Figure 3 , the environment has a set of acoustic characteristics associated with the environment. The audio system identifies the acoustic characteristics by estimating an impulse response in the environment at the user's position in the environment. The audio system can estimate the impulse response in the user's current environment by running a controlled measurement using an audio test signal generated by a mobile device or an impulse audio signal generated by the user (e.g., clapping). For example, in one embodiment, the audio system can estimate the impulse response using a measurement of the room reverberation time. Alternatively, the audio system can use sensor data and machine learning to determine room parameters and determine the impulse response accordingly. The impulse response in the user's current environment is stored as a raw response.

[0048] The audio system receives 420 a selection of a target environment from the user. The audio system may present a database of available target environment options to the user, allowing the user to select a particular room, hall, stadium, etc. In one embodiment, the target environment may be determined by the game engine based on a game scenario (e.g., a user enters a large, quiet church with marble floors). Each target environment option is associated with a set of target acoustic characteristics, which may also be stored with the database of available target environment options. For example, the target acoustic characteristics of a quiet church with marble floors may include echoes. The audio system characterizes the target acoustic characteristics by determining a target response.

[0049] The audio system receives 430 audio content from the user's environment. The audio content may be generated by the user of the audio system or ambient noise in the environment. An array of sensors in the audio system detects the sound. As described above, one or more sources of interest (e.g., the user's mouth, a musical instrument, etc.) may be tracked using DOA estimation, video tracking, beamforming, etc.

[0050] The audio system determines 440 a transfer function by comparing the acoustic characteristics of the user's current environment with the acoustic characteristics of a target environment. The acoustic characteristics of the current environment are characterized by an original response, while the acoustic characteristics of the target environment are characterized by a target response. The transfer function may be generated using real-time simulation, a database of measured responses, or an algorithmic reverberation method. Thus, the audio system adjusts 450 the detected audio content based on the target acoustic characteristics of the target environment. In one embodiment, as Figure 3 As described, the audio system convolves the transfer function with the audio content to generate a convolved audio signal. The audio system can utilize sound filters to amplify, attenuate, or enhance the detected sound.

[0051] The audio system renders 460 the adjusted audio content and presents it to the user through the speaker array. The adjusted audio content has at least some target acoustic properties so that the user perceives the sounds as if they are in the target environment.

[0052] Examples of Artificial Reality Systems

[0053] Figure 5 is a block diagram of an example artificial reality system 500 according to one or more embodiments. Artificial reality system 500 presents an artificial reality environment, such as a virtual reality, augmented reality, mixed reality environment, or some combination thereof, to a user. System 500 includes a near eye display (NED) 505 and an input / output (I / O) interface 555, both coupled to a console 510, and near eye display (NED) 505 may include a head mounted device and / or a head mounted display (HMD). System 500 also includes a mapping server 570 coupled to a network 575. Network 575 is coupled to NED 505 and console 510. NED 505 may be an embodiment of head mounted device 100. Although Figure 5 An example system is shown having one NED, one console, and one I / O interface, but in other embodiments any number of these components may be included in system 500 .

[0054] The NED 505 presents content to the user, which includes an augmented view of a physical, real-world environment with computer-generated elements (e.g., two-dimensional (2D) or three-dimensional (3D) images, 2D or 3D video, sound, etc.). The NED 505 can be an eyewear device or a head-mounted display. In some embodiments, the presented content includes audio content presented via an audio system 300, which receives audio information (e.g., an audio signal) from the NED 505, the console 510, or both, and presents the audio content based on the audio information. The NED 505 presents artificial reality content to the user. The NED includes an audio system 300, a depth camera assembly (DCA) 530, an electronic display 535, an optical block 540, one or more position sensors 545, and an inertial measurement unit (IMU) 550. The position sensor 545 and the IMU 550 are embodiments of sensors 140A-140B. In some embodiments, the NED 505 includes components other than those described herein. In addition, the functional distribution of the various components may be different from what is described herein.

[0055] The audio system 300 provides audio content to a user of the NED 505. As described above, reference Figure 1-4 , the audio system 300 renders audio content for a target artificial reality environment. The sensor array 310 captures the audio content, and the controller 330 analyzes the audio content for the acoustic characteristics of the environment. Using the acoustic characteristics of the environment and a set of target acoustic characteristics of the target environment, the controller 330 determines a transfer function. The transfer function is convolved with the detected audio content, resulting in the adjusted audio content having at least some of the acoustic characteristics of the target environment. The speaker array 320 presents the adjusted audio content to the user, presenting the sound as if it were being transmitted in the target environment.

[0056] The DCA 530 captures data describing depth information of the local environment surrounding some or all of the NED 505. The DCA 530 may include a light generator (e.g., structured light and / or flash for time of flight), an imaging device, and a DCA controller that may be coupled to both the light generator and the imaging device. The light generator illuminates the local area with illumination light, for example, according to a firing instruction generated by the DCA controller. The DCA controller is configured to control the operation of specific components of the light generator based on the firing instruction, for example, to adjust the intensity and pattern of the illumination light that illuminates the local area. In some embodiments, the illumination light may include a structured light pattern, such as a dot pattern, a line pattern, etc. The imaging device captures one or more images of one or more objects in the local area illuminated with the illumination light. The DCA 530 may use the data captured by the imaging device to calculate the depth information, or the DCA 530 may send the information to another device (e.g., the console 510), which may use the data from the DCA 530 to determine the depth information.

[0057] In some embodiments, the audio system 300 can utilize depth information obtained from the DCA 530. The audio system 300 can use the depth information to identify the direction of one or more potential sound sources, the depth of one or more sound sources, the movement of one or more sound sources, the sound activity around one or more sound sources, or any combination thereof. In some embodiments, the audio system 300 can use the depth information from the DCA 530 to determine acoustic parameters of the user's environment.

[0058] The electronic display 535 displays 2D or 3D images to the user based on the data received from the console 510. In various embodiments, the electronic display 535 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of the electronic display 535 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof. In some embodiments, the electronic display 535 displays visual content associated with the audio content presented by the audio system 300. When the audio system 300 presents the audio content adjusted to sound as if it were presented in the target environment, the electronic display 535 can present visual content depicting the target environment to the user.

[0059] In some embodiments, the optical block 540 amplifies image light received from the electronic display 535, corrects optical errors associated with the image light, and presents the corrected image light to a user of the NED 505. In various embodiments, the optical block 540 includes one or more optical elements. Example optical elements included in the optical block 540 include: a waveguide, an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 540 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 540 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0060] The magnification and focusing of the image light by the optical block 540 allows the electronic display 535 to be physically smaller, lighter, and consume less power than a larger display. In addition, the magnification can increase the field of view of the content presented by the electronic display 535. For example, the field of view of the displayed content is such that the displayed content uses almost all of the user's field of view (e.g., about 110 degrees diagonal), and in some cases, uses all of the user's field of view to be presented. In addition, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0061] In some embodiments, the optical block 540 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display 535 for display is pre-distorted, and when the optical block 540 receives image light generated based on the content from the electronic display 535, the optical block 540 corrects the distortion.

[0062] The IMU 550 is an electronic device that generates data indicating the position of the NED 505 based on measurement signals received from one or more position sensors 545. The position sensor 545 generates one or more measurement signals in response to the movement of the NED 505. Examples of the position sensor 545 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 550, or some combination thereof. The position sensor 545 can be located outside the IMU 550, inside the IMU 550, or some combination of these two locations. In one or more embodiments, the IMU 550 and / or the position sensor 545 can be a sensor in the sensor array 310, which is configured to capture data about the audio content presented by the audio system 300.

[0063] Based on one or more measurement signals from one or more position sensors 545, the IMU 550 generates data indicating an estimated current position of the NED 505 relative to an initial position of the NED 505. For example, the position sensors 545 include a plurality of accelerometers for measuring translational motion (forward / backward, up / down, left / right) and a plurality of gyroscopes for measuring rotational motion (e.g., pitch, yaw, and roll). In some embodiments, the IMU 550 samples the measurement signals rapidly and calculates the estimated current position of the NED 505 based on the sampled data. For example, the IMU 550 integrates the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated current position of a reference point on the NED 505. Alternatively, the IMU 550 provides the sampled measurement signals to the console 510, which parses the data to reduce errors. A reference point is a point that can be used to describe the position of the NED 505. A reference point may generally be defined as a point in space or a location relative to the orientation and position of the NED 505 .

[0064] The I / O interface 555 is a device that allows a user to send an action request and receive a response from the console 510. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end capturing image or video data, or an instruction to perform a specific action within an application. The I / O interface 555 may include one or more input devices. Example input devices include: a keyboard, a mouse, a hand controller, or any other suitable device for receiving an action request and transmitting the action request to the console 510. The action request received by the I / O interface 555 is transmitted to the console 510, and the console 510 performs the action corresponding to the action request. In some embodiments, the I / O interface 555 includes an IMU 550, as further described above, which captures calibration data indicating the estimated position of the I / O interface 555 relative to the initial position of the I / O interface 555. In some embodiments, the I / O interface 555 can provide tactile feedback to the user according to the instructions received from the console 510. For example, tactile feedback is provided when an action request is received, or when console 510 transmits an instruction to I / O interface 555 causing I / O interface 555 to generate tactile feedback when console 510 performs an action. I / O interface 555 may monitor one or more input responses from a user for determining a perceived source direction and / or perceived source position of audio content.

[0065] Console 510 provides content to NED 505 for processing based on information received from one or more of: NED 505 and I / O interface 555. Figure 5 In the example shown, the console 510 includes an application store 520, a tracking module 525, and an engine 515. Some embodiments of the console 510 have a Figure 5 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Figure 5 The described approach is distributed among the components of console 510 .

[0066] Application storage 520 stores one or more applications for execution by console 510. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be responsive to input received from a user via a mobile or I / O interface 555 of NED 505. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0067] The tracking module 525 calibrates the system environment 500 using one or more calibration parameters, and may adjust the one or more calibration parameters to reduce errors in the position determination of the NED 505 or the I / O interface 555. The calibration performed by the tracking module 525 may also take into account information received from the IMU 550 in the NED 505 and / or the IMU 550 included in the I / O interface 555. In addition, if tracking of the NED 505 is lost, the tracking module 525 may recalibrate some or all of the system environment 500.

[0068] The tracking module 525 tracks the movement of the NED 505 or the I / O interface 555 using information from one or more position sensors 545, the IMU 550, the DCA 530, or some combination thereof. For example, the tracking module 525 determines the location of a reference point of the NED 505 in a map of the local area based on the information from the NED 505. The tracking module 525 may also determine the location of the reference point of the NED 505 or the reference point of the I / O interface 555 using data from the IMU 550 indicating the location of the NED 505 or using data from the IMU 550 included in the I / O interface 555 indicating the location of the I / O interface 555, respectively. Furthermore, in some embodiments, the tracking module 525 may use a portion of the data from the IMU 550 indicating the location of the NED 505 to predict the future location of the NED 505. The tracking module 525 provides the estimated or predicted future location of the NED 505 or the I / O interface 555 to the engine 515. In some embodiments, tracking module 525 may provide tracking information to audio system 300 for use in generating sound filters.

[0069] The engine 515 also executes applications within the system environment 500 and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the NED 505 from the tracking module 525. Based on the received information, the engine 515 determines the content to be provided to the NED 505 for presentation to the user. For example, if the received information indicates that the user is looking to the left, the engine 515 generates content for the NED 505 that reflects the user's movement in a virtual environment or in an environment that enhances a local area with additional content. In addition, the engine 515 performs actions within an application executed on the console 510 in response to action requests received from the I / O interface 555, and provides feedback to the user that the action has been performed. The feedback provided can be visual or auditory feedback via the NED 505 or tactile feedback via the I / O interface 555.

[0070] The mapping server 570 can provide audio and visual content to the NED 505 for presentation to the user. The mapping server 570 includes a database that stores virtual models describing multiple environments and acoustic characteristics of these environments (including multiple target environments and their associated acoustic characteristics). The NED 505 can query the mapping server 570 for the acoustic characteristics of the environment. The mapping server 570 receives visual information describing at least a portion of the environment (e.g., a room) in which the user is currently located and / or location information of the NED 505 from the NED 505 via the network 575. The mapping server 570 determines a location in the virtual model associated with the current configuration of the room based on the received visual information and / or location information. The mapping server 570 determines (e.g., retrieves) a set of acoustic parameters associated with the current configuration of the room based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The mapping server 570 can also receive information about the target environment that the user wants to simulate through the NED 505. The mapping server 570 determines (e.g., retrieves) a set of acoustic parameters associated with the target environment. The mapping server 570 may provide information about a set of acoustic parameters, about the user's current environment, and / or a target environment to the NED 505 (e.g., via the network 575) for use in generating audio content at the NED 505. Alternatively, the mapping server 570 may generate an audio signal using the set of acoustic parameters and provide the audio signal to the NED 505 for rendering. In some embodiments, some components of the mapping server 570 may be integrated with another device (e.g., the console 510) connected to the NED 505 via a wired connection.

[0071] The network 575 connects the NED 505 to the mapping server 570. The network 575 may include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, the network 575 may include the Internet as well as a mobile phone network. In one embodiment, the network 575 uses standard communication technologies and / or protocols. Therefore, the network 575 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 575 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. The data exchanged through the network 575 may be represented using technologies and / or formats including image data in binary form (e.g., Portable Network Graphics (PNG)), Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, all or some of the links may be encrypted using conventional encryption techniques such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. The network 575 may also connect multiple head mounted devices located in the same or different rooms to the same mapping server 570. The use of mapping servers and networks to provide audio and visual content is described in more detail in U.S. Patent Application No. 16 / 366,484 filed on March 27, 2019, which is incorporated herein in its entirety by this application.

[0072] Additional configuration information

[0073] The foregoing description of the embodiments of the present disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art may recognize that many modifications and variations are possible in light of the above disclosure.

[0074] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. These operations (although described functionally, computationally or logically) should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. associated with the manufacturing process. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0075] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations or processes described (e.g., with respect to a manufacturing process).

[0076] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-temporary, tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0077] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the inventive subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the disclosure, which is set forth in the appended claims.

Claims

1. A method for rendering sound in a target artificial reality environment, comprising: Analyzing sounds in a user's current environment (200) to identify a set of acoustic characteristics associated with the user's current environment (200); Receiving audio content generated in the user's current environment (200); determining a transfer function based on a comparison of the set of acoustic characteristics to a set of target acoustic characteristics of a target artificial reality environment; adjusting the audio content using the transfer function, wherein the transfer function adjusts the set of acoustic characteristics of the audio content based on the set of target acoustic characteristics of the target artificial reality environment; and The adjusted audio content is presented to a user, wherein the adjusted audio content is perceived by the user as being generated in the target artificial reality environment, the method further comprising: providing a plurality of target artificial reality environment options to the user, each of the plurality of target artificial reality environment options corresponding to a different target artificial reality environment; and receiving from the user a selection of the target artificial reality environment from the plurality of target artificial reality environment options.

2. The method according to claim 1, wherein: Adjusting the audio content using the transfer function further comprises: identifying ambient sounds in the user's current environment (200); and The environmental sound is filtered out from the adjusted audio content for the user.

3. The method according to claim 1, wherein: Each of the plurality of target artificial reality environment options is associated with a different set of acoustic characteristics of the target artificial reality environment.

4. The method according to claim 1, further comprising: determining a raw response characterizing the set of acoustic characteristics associated with the user's current environment (200); and A target response is determined that characterizes the set of target acoustic properties of the target artificial reality environment.

5. The method according to claim 4, wherein: Determining the transfer function further comprises: comparing the original response and the target response; and Based on the comparison, a difference between a set of acoustic parameters associated with the user's current environment (200) and a set of acoustic parameters associated with the target artificial reality environment is determined.

6. The method according to claim 1, further comprising: An acoustic filter is generated using the transfer function, wherein the adjusted audio content is based in part on the acoustic filter.

7. The method according to claim 1, wherein: Determining the transfer function is based on at least one previously measured room impulse or algorithmic reverberation.

8. The method according to claim 1, wherein: Adjusting the audio content further includes: The transfer function is convolved with the received audio content.

9. The method according to claim 1, wherein: The received audio content is generated by at least one user among the plurality of users.

10. An audio system comprising: one or more sensors configured to receive audio content in the user's current environment (200); one or more speakers configured to present audio content to a user; and A controller configured to: Analyzing sounds in the user's current environment (200) to identify a set of acoustic characteristics associated with the user's current environment (200); determining a transfer function based on a comparison of the set of acoustic characteristics to a set of target acoustic characteristics of a target artificial reality environment; adjusting the audio content using the transfer function, wherein the transfer function adjusts the set of acoustic characteristics of the audio content based on the set of target acoustic characteristics of the target artificial reality environment; and instructing the speaker to present the adjusted audio content to a user, wherein the adjusted audio content is perceived by the user as being generated in the target artificial reality environment, Wherein, the controller is further configured as: providing a plurality of target artificial reality environment options to a user, each of the plurality of target artificial reality environment options corresponding to a different target artificial reality environment; and A selection of the target artificial reality environment from among the plurality of target artificial reality environment options is received from a user.

11. The system according to claim 10, wherein: The audio system is part of a head mounted device.

12. The system according to claim 10, wherein: Adjusting the audio content further includes: identifying ambient sounds in the user's current environment (200); and The environmental sound is filtered out from the adjusted audio content for the user.

13. The system according to claim 10, wherein: Each of the plurality of target artificial reality environment options is associated with a set of target acoustic characteristics of the target artificial reality environment.

14. The system according to claim 10, wherein: The controller is also configured to: determining a raw response characterizing the set of acoustic characteristics associated with the user's current environment (200); and A target response is determined that characterizes the set of target acoustic properties of the target artificial reality environment.

15. The system of claim 14, wherein: The controller is also configured to: A room impulse response of the user's current environment (200) is estimated, wherein the room impulse response is used to generate the original response.

16. The system of claim 10, wherein: The controller is also configured to: generating a sound filter using the transfer function; and The audio content is adjusted based in part on the sound filter.

17. The system according to claim 10, wherein: The controller is also configured to: The transfer function is determined using at least one previously measured room impulse response or algorithmic reverberation.

18. The system of claim 10, wherein: The controller is configured to adjust the audio content by convolving the transfer function with the received audio content.

Images