Environmental acoustic persistence

Abstract

Systems and methods for storing, organizing, and maintaining acoustic data for a mixed reality system are disclosed herein. A system can include one or more sensors of a head-mounted device, a speaker of the head-mounted device, and one or more processors configured to perform a method. A method for execution by the one or more processors can include receiving a request to present an audio signal. An environment can be identified via the one or more sensors of the head-mounted device. One or more audio model components associated with the environment can be retrieved. A first audio model can be generated based on the audio model components. A second audio model can be generated based on the first audio model. A modified audio signal can be determined based on the second audio model and based on the request to present the audio signal. The modified audio signal can be presented via the speaker of the head-mounted device.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 944,956, filed December 6, 2019, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to systems and methods for managing and storing audio data, and particularly to systems and methods for managing and storing audio data in mixed reality environments. Background Technology

[0004] Virtual environments are ubiquitous in computing environments, finding applications in video games (where a virtual environment can represent a game world); maps (where a virtual environment can represent terrain to navigate); simulations (where a virtual environment can simulate a real environment); digital storytelling (where virtual characters can interact with each other within a virtual environment); and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, the user experience with virtual environments can be limited by the technologies used to render them. For example, traditional displays (e.g., 2D screens) and audio systems (e.g., fixed speakers) may not be able to render virtual environments in a way that creates a compelling, realistic, and immersive experience.

[0005] Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively, “XR”) share the ability to present users of XR systems with sensory information corresponding to a virtual environment represented by data from a computer system. Such systems can provide uniquely enhanced immersion and realism by combining virtual visual and audio cues with real-world sight and sound. Therefore, it may be desirable to present digital sound to users of XR systems in such a way that the sound appears natural and consistent with the user’s expectations of sound in their real environment. Generally, users expect virtual sound to have the acoustic characteristics of their real environment. For example, users of an XR system in a large concert hall would expect the virtual sound to have a large, spacious, and hollow quality; conversely, users in a small apartment would expect the sound to be more attenuated, closer, and more immediate. In addition to matching virtual sound to the acoustic characteristics of real and / or virtual environments, the sense of realism is further enhanced by spatializing virtual sound. For example, a virtual object might visually fly past the user from behind, and the user might expect the corresponding virtual sound to similarly reflect the virtual object’s spatial movement relative to the user.

[0006] Existing technologies often fall short of these expectations, such as by presenting virtual audio that disregards the user's surroundings or the spatial movement of virtual objects, leading to an unrealistic experience that can harm the user experience. Observations on XR system users suggest that while users may be relatively tolerant of visual mismatches between virtual content and the real environment (e.g., inconsistent lighting), they may be more sensitive to auditory mismatches. Our own auditory experience, refined throughout our lives, makes us acutely aware of how our physical environment affects the sounds we hear; we can be highly perceptive of sounds that are inconsistent with expectations. With XR systems, such inconsistencies can be jarring and can turn an immersive and engaging experience into a clunky imitation. In extreme cases, auditory inconsistencies can lead to motion sickness and other adverse effects because the inner ear cannot coordinate auditory stimuli with their corresponding visual cues.

[0007] A system architecture is needed to organize and manage the systems used to generate virtual audio. Generating virtual audio may involve managing and storing information about the user's environment so that this information can be used to generate realistic virtual sounds. Therefore, the audio system architecture may need to interface with other systems to receive and utilize information related to the audio engine. An audio system architecture that can deliver realistic sound without interruption during use may also be desired. An audio system architecture that can update the audio engine without interrupting the user can produce an immersive user experience, where auditory signals continuously reflect the user's environment.

[0008] By considering the characteristics of the user's physical environment, the systems and methods described in this paper can simulate what a user would hear if the virtual sound were a real sound naturally generated in that environment. By presenting virtual sound in a way that faithfully reflects how sound behaves in the real world, users can experience a high degree of engagement with the mixed reality environment. Similarly, by presenting location-aware virtual content that responds to user movement and the environment, the content becomes more subjective, interactive, and realistic—for example, a user's experience at point A may be completely different from their experience at point B. This enhanced realism and interactivity can provide the foundation for new mixed reality applications, such as applications using spatially aware audio enabling novel forms of gameplay, social features, or interactive behaviors. Summary of the Invention

[0009] This document discloses systems and methods for storing, organizing, and maintaining acoustic data for mixed reality systems. A system may include one or more sensors of a head-mounted device, speakers of the head-mounted device, and one or more processors configured to perform a method. The method, executed by the one or more processors, may include receiving a request to present an audio signal. An environment may be identified via one or more sensors of the head-mounted device. One or more audio model components associated with the environment may be retrieved. A first audio model may be generated based on the audio model components. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and the request to present the audio signal. The modified audio signal may be presented via speakers of the head-mounted device.

[0010] In some system embodiments, the second audio model may be generated by an audio service. In some system embodiments, the modified audio signal may be determined by an audio service. In some system embodiments, the second audio model may be a copy of the first audio model.

[0011] In some system embodiments, one or more audio model components may include one or more dimensions of the environment. In some system embodiments, one or more audio model components may include reverberation time. In some system embodiments, one or more audio model components may include reverberation gain. In some system embodiments, one or more audio model components may include transmission loss factors. In some system embodiments, one or more audio model components may include absorption factors.

[0012] A method may include receiving a request to present an audio signal. The environment may be identified via one or more sensors of a head-mounted device. One or more audio model components associated with the environment may be retrieved. A first audio model based on the audio model components may be generated. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and the request to present the audio signal. The modified audio signal may be presented via a speaker of the head-mounted device.

[0013] In some implementations, the second audio model may be generated by an audio service. In some implementations, the modified audio signal may be determined by an audio service. In some implementations, the second audio model may be a copy of the first audio model.

[0014] In some method embodiments, one or more audio model components may include one or more dimensions of the environment. In some method embodiments, one or more audio model components may include reverberation time. In some method embodiments, one or more audio model components may include reverberation gain. In some method embodiments, one or more audio model components may include transmission loss coefficients. In some method embodiments, one or more audio model components may include absorption coefficients.

[0015] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method, to be executed by one or more processors, may include: receiving a request to present an audio signal; identifying an environment via one or more sensors of a head-mounted device; retrieving one or more audio model components associated with the environment; generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and the request to present the audio signal; and presenting the modified audio signal via a speaker of the head-mounted device.

[0016] In some non-transitory computer-readable medium embodiments, the second audio model may be generated by an audio service. In some non-transitory computer-readable medium embodiments, the modified audio signal may be determined by an audio service. In some non-transitory computer-readable medium embodiments, the second audio model may be a copy of the first audio model.

[0017] In some non-transitory computer-readable medium embodiments, one or more audio model components may include one or more dimensions of the environment. In some non-transitory computer-readable medium embodiments, one or more audio model components may include reverberation time. In some non-transitory computer-readable medium embodiments, one or more audio model components may include reverberation gain. In some non-transitory computer-readable medium embodiments, one or more audio model components may include transmission loss factors. In some non-transitory computer-readable medium embodiments, one or more audio model components may include absorption factors. Attached Figure Description

[0018] Figures 1A to 1C An example mixed reality environment is shown according to some embodiments.

[0019] Figures 2A to 2D Components of an example mixed reality system, according to some embodiments, are shown that can be used to generate and interact with mixed reality environments.

[0020] Figure 3A An example mixed reality handheld controller, according to some embodiments, is shown that can be used to provide input to a mixed reality environment.

[0021] Figure 3B An example auxiliary unit, which can be used with an example mixed reality system according to some embodiments, is shown.

[0022] Figure 4 An example functional block diagram for an example mixed reality system is shown according to some embodiments.

[0023] Figure 5 An example of a virtual audio system according to some embodiments is shown.

[0024] Figure 6 An example process for updating an audio model is shown according to some embodiments.

[0025] Figure 7 An example process for updating an audio model is shown according to some embodiments. Detailed Implementation

[0026] In the following description of the examples, reference is made to the accompanying drawings, which form part of the drawings and illustrate, by way of illustration, examples that can be practiced. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0027] Mixed Reality Environment

[0028] Like everyone else, users of mixed reality systems exist within a real-world environment—that is, the three-dimensional portion of the “real world” and all its content that can be perceived by the user. For example, users use ordinary human senses to perceive the real-world environment—sight, sound, touch, taste, smell—and interact with it by moving their own bodies within it. A location in the real-world environment can be described as coordinates in a coordinate space; for example, coordinates could include latitude, longitude, and altitude relative to sea level; distance from a reference point in three orthogonal dimensions; or other suitable values. Similarly, a vector can describe a quantity having direction and magnitude in coordinate space.

[0029] Computing devices may maintain a representation of a virtual environment, for example, in memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment may include a representation of any object, action, signal, parameter, coordinate, vector, or other characteristic associated with that space. In some examples, the circuitry of the computing device (e.g., a processor) may maintain and update the state of the virtual environment; that is, the processor may determine the state of the virtual environment at a second time t1 at a first time t0 based on data associated with the virtual environment and / or input provided by the user. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has some programmed physical parameter (e.g., mass, coefficient of friction); and input received from the user indicates that a force should be applied to the object in a direction vector; the processor may apply laws of kinematics to determine the position of the object at time t1 using fundamental mechanics. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. While maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to the creation and deletion of virtual objects in the virtual environment; software for defining the behavior of virtual objects or characters in the virtual environment (e.g., scripts); software for defining the behavior of signals (e.g., audio signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating audio signals in the virtual environment; software for processing inputs and outputs; software for implementing network operations; software for applying asset data (e.g., animation data of virtual objects moving over time); or many other possibilities.

[0030] Output devices (such as displays or speakers) can present any or all aspects of a virtual environment to a user. For example, a virtual environment may include virtual objects that can be presented to the user (which may include representations of inanimate objects; people; animals; light, etc.). A processor can determine a view of the virtual environment (e.g., corresponding to a “camera” with a coordinate origin, view axes, and view frustum); and render a visual scene of the virtual environment corresponding to that view to the display. Any suitable rendering technique can be used for this purpose. In some examples, the visual scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, a virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, virtual objects in the virtual environment may generate sounds originating from the object's location coordinates (e.g., a virtual character can speak or cause sound effects); or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a specific location. The processor can determine the audio signal corresponding to the “listener” coordinates—for example, a composite audio signal corresponding to sounds in a virtual environment, and mix and process it to simulate the audio signal that will be heard by the listener at the listener coordinates—and present the audio signal to the user via one or more speakers.

[0031] Because the virtual environment exists solely as a computational construct, users may not directly perceive it using their ordinary senses. Instead, users can only perceive it indirectly, such as through displays, speakers, haptic output devices, etc. Similarly, users may not directly touch, manipulate, or otherwise interact with the virtual environment; however, they can provide input data to a processor that can update the virtual environment using this data via input devices or sensors. For example, a camera sensor can provide optical data indicating that a user is attempting to move an object in the virtual environment, and the processor can use this data to cause the object to respond accordingly within the virtual environment.

[0032] Mixed reality systems can present a mixed reality environment (“MRE”) to a user, combining aspects of a real and a virtual environment, for example, using a transmissive display and / or one or more speakers (which may be incorporated, for example, into a wearable head-mounted device). In some embodiments, one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space; in some examples, the real coordinate space and the corresponding virtual coordinate space are correlated with each other through a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with the transformation matrix) can define a first position in the real environment and a second corresponding position in the virtual environment; and vice versa.

[0033] In MRE, virtual objects (e.g., in a virtual environment associated with the MRE) can correspond to real objects (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamppost (real object) at location coordinates, then the virtual environment of the MRE can include a virtual lamppost (virtual object) at the corresponding location coordinates. As used herein, real objects combined with their corresponding virtual objects constitute "mixed reality objects." Virtual objects do not need to perfectly match or align with their corresponding real objects. In some examples, virtual objects can be simplified versions of their corresponding real objects. For example, if the real environment includes a real lamppost, the corresponding virtual object can include a cylinder with roughly the same height and radius as the real lamppost (reflecting that the lamppost can be roughly cylindrical in shape). Simplifying virtual objects in this way allows for computational efficiency and simplifies the computations performed on such virtual objects. Furthermore, in some examples of the MRE, not all real objects in the real environment can be associated with corresponding virtual objects. Similarly, in some examples of the MRE, not all virtual objects in the virtual environment can be associated with corresponding real objects. That is, some virtual objects can exist only in the virtual environment of MRE without any real-world counterparts.

[0034] In some examples, virtual objects can possess characteristics that differ (sometimes drastically) from their corresponding real-world counterparts. For instance, while a real-world environment in an MRE might include a green, two-armed cactus—a thorny, inanimate object—the corresponding virtual object in the MRE could possess characteristics of a green, two-armed virtual character with human facial features and aggressive behavior. In this example, the virtual object resembles its real-world counterpart in some characteristics (color, number of arms); but differs from the real object in other characteristics (facial features, personality). In this way, virtual objects have the potential to represent real-world objects in a creative, abstract, exaggerated, or imaginative way; or to impart behavior (e.g., human personality) to other inanimate real-world objects. In some examples, virtual objects can be purely imaginary creations without any real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps occupying a location corresponding to an empty space in a real environment).

[0035] Compared to VR systems, which present a virtual environment to the user while blurring the real environment, mixed reality (MRE) systems offer the advantage of maintaining the perceptibility of the real environment while the virtual environment is presented. Therefore, users of MRE systems can experience and interact with the corresponding virtual environment using visual and audio cues associated with the real environment. For example, while a VR system user might strive to perceive or interact with virtual objects displayed in the virtual environment—because, as mentioned above, the user cannot directly perceive or interact with the virtual environment—an MR system user can find interacting with virtual objects intuitive and natural by seeing, hearing, and touching corresponding real objects in their own real environment. This level of interactivity can enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by simultaneously presenting the real and virtual environments, MRE systems can reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Furthermore, MRE systems offer numerous possibilities for applications that can enhance or alter our experience of the real world.

[0036] Figure 1A An exemplary real-world environment 100 is shown where user 110 uses a mixed reality system 112. The mixed reality system 112 may include a display (e.g., a transmissive display) and one or more speakers, as well as one or more sensors (e.g., cameras), such as those described below. The real-world environment 100 shown includes a rectangular room 104A where user 110 stands; and real-world objects 122A (lamp), 124A (table), 126A (sofa), and 128A (painting). Room 104A also includes positional coordinates 106, which can be considered the origin of the real-world environment 100. Figure 1AAs shown, an environment / world coordinate system 108 (including x-axis 108X, y-axis 108Y, and z-axis 108Z) with its origin at point 106 (world coordinates) can define the coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 can correspond to the location where the mixed reality environment 112 is powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 can be reset during operation. In some examples, the user 110 can be considered a real object in the real environment 100; similarly, body parts of the user 110 (e.g., hands, feet) can be considered real objects in the real environment 100. In some examples, a user / audience / head coordinate system 114 (including x-axis 114X, y-axis 114Y, and z-axis 114Z) with its origin at point 115 (e.g., user / audience / head coordinates) can define the coordinate space for the user / audience / head where the mixed reality system 112 is located. The origin 115 of the user / audience / head coordinate system 114 can be defined relative to one or more components of the mixed reality system 112. For example, the origin 115 of the user / audience / head coordinate system 114 can be defined relative to the display of the mixed reality system 112, such as during the initial calibration of the mixed reality system 112. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the user / audience / head coordinate system 114 space and the environment / world coordinate system 108 space. In some embodiments, left ear coordinates 116 and right ear coordinates 117 can be defined relative to the origin 115 of the user / audience / head coordinate system 114. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / audience / head coordinate system 114 space. The user / audience / head coordinate system 114 can simplify the representation of position relative to the user's head or head-mounted device (e.g., relative to the environment / world coordinate system 108). Using simultaneous localization and mapping (SLAM), visual ranging, or other techniques, the transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time.

[0037] Figure 1BAn exemplary virtual environment 130 corresponding to the real environment 100 is shown. The virtual environment 130 shown includes a virtual rectangular room 104B corresponding to the real rectangular room 104A; a virtual object 122B corresponding to the real object 122A; a virtual object 124B corresponding to the real object 124A; and a virtual object 126B corresponding to the real object 126A. Metadata associated with the virtual objects 122B, 124B, and 126B may include information derived from the corresponding real objects 122A, 124A, and 126A. The virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in the real environment 100. A real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate frame 133 (including x-axis 133X, y-axis 133Y, and z-axis 133Z) with its origin at point 134 (persistent coordinates) can define the coordinate space used for the virtual content. The origin 134 of the persistent coordinate frame 133 can be defined relative to / about one or more real objects (such as real object 126A). Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the space of the persistent coordinate frame 133 and the environment / world coordinate system 108 space. In some embodiments, each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to the origin 134 of the persistent coordinate frame 133. In some embodiments, multiple persistent coordinate frames can exist, and each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to one or more persistent coordinate frames.

[0038] Compared to Figure 1A and Figure 1B The environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the example shown, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Therefore, a first position in the real environment 100 and a second corresponding position in the virtual environment 130 can be described relative to the same coordinate system. This simplifies the identification and display of corresponding positions in the real and virtual environments, as the same coordinates can be used to identify both positions. However, in some examples, the corresponding real and virtual environments do not need to use a shared coordinate space. For example, in some examples (not shown), matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0039] Figure 1CAn exemplary MRE 150 is shown that simultaneously presents aspects of a real environment 100 and a virtual environment 130 to a user 110 via a mixed reality system 112. In the example shown, the MRE 150 simultaneously presents to the user 110 real objects 122A, 124A, 126A, and 128A from the real environment 100 (e.g., via the transmissive portion of the display of the mixed reality system 112); and virtual objects 122B, 124B, 126B, and 132 from the virtual environment 130 (e.g., via the active display portion of the display of the mixed reality system 112). As described above, the origin 106 serves as the origin for the coordinate space corresponding to the MRE 150, and the coordinate system 108 defines the x-axis, y-axis, and z-axis for the coordinate space.

[0040] In the illustrated example, the mixed reality objects include corresponding real and virtual objects occupying corresponding positions in coordinate space 108 (i.e., 122A / 122B, 124A / 124B, 126A / 126B). In some examples, both the real and virtual objects can be simultaneously visible to the user 110. This is desirable in instances where the virtual object presents information designed to enhance the view of the corresponding real object (such as in a museum application where the virtual object presents a missing piece of an ancient damaged sculpture). In some examples, the virtual object (122B, 124B, and / or 126B) can be displayed (e.g., via active pixelation occlusion using a pixelation occlusion shutter) to occlude the corresponding real object (122A, 124A, and / or 126A). This is desirable in instances where the virtual object acts as a visual replacement for the corresponding real object (such as in an interactive story application where an inanimate real object becomes a "living" character).

[0041] In some examples, real-world objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data that may not necessarily constitute a virtual object. Virtual content or helper data can facilitate the processing or disposal of virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of: the corresponding real-world object; a custom asset type associated with the corresponding real-world object; or statistical data associated with the corresponding real-world object. This information enables or facilitates computations involving real-world objects without incurring unnecessary computational overhead.

[0042] In some examples, the presentation described above may also include audio aspects. For example, in MRE 150, the virtual monster 132 may be associated with one or more audio signals, such as footsteps generated as the monster walks around MRE 150. As further described below, the processor of Mixed Reality System 112 may calculate the audio signal corresponding to the mixture and processing of all such sounds in MRE 150, and present the audio signal to user 110 via one or more speakers included in Mixed Reality System 112 and / or one or more external speakers.

[0043] Example Mixed Reality System

[0044] Example mixed reality system 112 may include a wearable head-mounted device (e.g., a wearable augmented reality or mixed reality head-mounted device) comprising: a display (which may include left and right transmissive displays, which may be near-eye displays, and associated components for coupling light from the displays to the user's eyes); left and right speakers (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measurement unit (IMU) (e.g., mounted to a side arm of the head-mounted device); an orthogonal coil electromagnetic receiver (e.g., mounted to the left side arm); left and right cameras (e.g., depth (time-of-flight) cameras) oriented away from the user; and left and right eye cameras oriented towards the user (e.g., for detecting the user's eye movements). However, mixed reality system 112 may incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LiDAR, EOG, GPS, magnetic). Furthermore, mixed reality system 112 may include networking features (e.g., Wi-Fi capability) for communicating with other devices and systems, including other mixed reality systems. The mixed reality system 112 may also include a battery (which may be housed in an auxiliary unit, such as a belt bag designed to be worn around the user's waist), a processor, and memory. The wearable head-mounted device of the mixed reality system 112 may include tracking components, such as an IMU or other suitable sensors, configured to output a set of coordinates of the wearable head-mounted device relative to the user's environment. In some examples, the tracking components may provide input to a processor performing simultaneous localization and mapping (SLAM) and / or visual ranging algorithms. In some examples, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt bag, as further described below.

[0045] Figures 2A to 2D Components of an exemplary mixed reality system 200 (which may correspond to mixed reality system 112) that can be used to present an MRE (which may correspond to MRE 150) or other virtual environment to a user are shown. Figure 2A A perspective view of a wearable head-mounted device 2102 included in an example mixed reality system 200 is shown. Figure 2B A top view of a wearable head device 2102 worn on a user's head 2202 is shown. Figure 2C A front view of the wearable head device 2102 is shown. Figure 2D A side view of an example eyepiece 2110 of a wearable head device 2102 is shown. Figures 2A to 2C As shown, the example wearable head device 2102 includes an exemplary left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 may include: a transmission element through which the real environment can be visible; and a display element for presenting a display superimposed on the real environment (e.g., via image-by-image modulated light). In some examples, such a display element may include surface diffraction optics for controlling the flow of image-by-image modulated light. For example, the left eyepiece 2108 may include a left-coupled-in grating set 2112, a left orthogonal pupil expansion (OPE) grating set 2120, and a left-exit (output) pupil expansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 may include a right-coupled-in grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. Image-by-image modulated light can be delivered to the user's eye via coupling gratings 2112 and 2118, OPEs 2114 and 2120, and EPEs 2116 and 2122. Each coupling grating set 2112, 2118 can be configured to deflect light toward its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to deflect light downwards incrementally toward its associated EPEs 2122, 2116, thereby horizontally extending the forming exit pupil. Each EPE 2122, 2116 can be configured to incrementally redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outwards to a user eyebox location (not shown) defined behind eyepieces 2108, 2110, vertically extending the exit pupil formed at the eyebox location. Alternatively, instead of coupling grating sets 2112 and 2118, OPE grating sets 2114 and 2120, and EPE grating sets 2116 and 2122, eyepieces 2108 and 2110 may include gratings and / or other arrangements for controlling the refractive and reflective characteristics of the image-by-image modulated light coupled to the user's eye.

[0046] In some examples, the wearable head device 2102 may include a left support arm 2130 and a right support arm 2132, wherein the left support arm 2130 includes a left speaker 2134 and the right support arm 2132 includes a right speaker 2136. An orthogonal coil electromagnetic receiver 2138 may be positioned in the left support arm or at another suitable location within the wearable head unit 2102. An inertial measurement unit (IMU) 2140 may be positioned in the right support arm 2132 or at another suitable location within the wearable head device 2102. The wearable head device 2102 may also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142 and 2144 may be suitably oriented in different directions to cover a wider field of view together.

[0047] exist Figures 2A to 2D In the example shown, the left source 2124 of the image-modulated light can be optically coupled to the left eyepiece 2108 via the left coupling grating set 2112, and the right source 2126 of the image-modulated light can be optically coupled to the right eyepiece 2110 via the right coupling grating set 2118. The image-modulated light sources 2124, 2126 may include, for example, fiber optic scanners; projectors including electro-optic modulators such as digital light processing (DLP) chips or liquid crystal on silicon (LCoS) modulators; or emitting displays such as micro-LED or micro-organic light-emitting diode (μOLED) panels, which are coupled to the coupling grating sets 2112, 2118 using one or more lenses on each side. The input coupling grating sets 2112, 2118 can deflect light from the image-modulated light sources 2124, 2126 to an angle greater than the critical angle for total internal reflection (TIR) ​​for the eyepieces 2108, 2110. OPE grating sets 2114 and 2120 incrementally deflect light propagating through TIR downwards toward EPE grating sets 2116 and 2122. EPE grating sets 2116 and 2122 incrementally couple light toward the user's face, including the pupils of the user's eyes.

[0048] In some examples, such as Figure 2DAs shown, each of the left eyepiece 2108 and the right eyepiece 2110 includes multiple waveguides 2402. For example, each eyepiece 2108, 2110 may include multiple individual waveguides, each dedicated to a corresponding color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 may include multiple sets of such waveguides, wherein each set is configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex relative to the user's eye, for example, to present a virtual object positioned at a distance in front of the user (e.g., by a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating sets 2116, 2122 may include curved grating recesses to achieve a convex wavefront curvature by varying the Poynting vector of the emitted light across each EPE.

[0049] In some examples, to create a perception that the displayed content is three-dimensional, stereoscopically accommodated left and right eye images can be presented to the user via image-by-image light modulators 2124, 2126 and eyepieces 2108, 2110. The perceived realism of the presentation of the three-dimensional virtual object can be enhanced by selecting waveguides (and therefore corresponding wavefront curvatures) so that the virtual object is displayed at a distance close to that indicated by the stereoscopic left and right images. This technique can also reduce motion sickness experienced by some users, which can be caused by the discrepancy between the depth perception cues provided by the stereoscopic left and right eye images and the autoacoustic function of the human eye (e.g., object distance-related focus).

[0050] Figure 2D An edge-facing view is shown from the top of the right eyepiece 2110 of the example wearable head device 2102. (See image.) Figure 2D As shown, multiple waveguides 2402 may include a first subset of three waveguides 2404 and a second subset of three waveguides 2406. The two subsets of waveguides 2404 and 2406 can be distinguished by different EPE gratings, each with a different line curvature to impart different wavefront curvatures to the emitted light. Within each subset of waveguides 2404 and 2406, each waveguide can be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. Although not shown in the diagram... Figure 2D However, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.

[0051] Figure 3AAn exemplary handheld controller component 300 of a mixed reality system 200 is shown. In some examples, the handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, the buttons 350 may be configured to serve as optical tracking targets, for example, for combining a camera or other optical sensor (which may be mounted in the head unit of the mixed reality system 200 (e.g., wearable head device 2102)) to track the six degrees of freedom (6DOF) motion of the handheld controller 300. In some examples, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting position or orientation (such as position or orientation relative to the wearable head device 2102). In some examples, such a tracking component may be positioned in the handle of the handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 may be configured to provide one or more output signals corresponding to one or more of the button press states; or the position, orientation, and / or movement of the handheld controller 300 (e.g., via an IMU). Such an output signal can be used as an input to the processor of the mixed reality system 200. Such input can correspond to the position, orientation, and / or movement of the handheld controller (e.g., by extension, to the position, orientation, and / or movement of the user's hand holding the controller). Such input can also correspond to the user pressing button 350.

[0052] Figure 3B An exemplary auxiliary unit 320 of a mixed reality system 200 is shown. The auxiliary unit 320 may include a battery that provides power to the operating system 200 and may include a processor for executing programs on the operating system 200. As shown, the example auxiliary unit 320 includes a chip 2128, such as for attaching the auxiliary unit 320 to a user's belt. Other form factors suited to the auxiliary unit 320 and will be apparent, including form factors not involving mounting the unit to the user's belt. In some examples, the auxiliary unit 320 is coupled to a wearable head device 2102 via a multi-conduit cable, which may include, for example, wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 may also be used.

[0053] In some examples, the mixed reality system 200 may include one or more microphones that detect sound and provide corresponding signals to the mixed reality system. In some examples, the microphone may be attached to or integrated with a wearable head device 2102 and configured to detect the user's voice. In some examples, the microphone may be attached to or integrated with a handheld controller 300 and / or auxiliary unit 320. Such a microphone may be configured to detect ambient sound, ambient noise, the user's or a third party's voice, or other sounds.

[0054] Figure 4 An exemplary functional block diagram is shown that can correspond to an exemplary mixed reality system, such as the mixed reality system 200 described above (which can correspond to the mixed reality system 112 relative to FIG1). Figure 4 As shown, the example handheld controller 400B (which may correspond to the handheld controller 300 (“Totem”)) includes a totem-to-wear head-mounted device six-DOF (6DOF) totem subsystem 404A, and the example wearable head-mounted device 400A (which may correspond to the wearable head-mounted device 2102) includes a totem-to-wearable head-mounted device 6DOF subsystem 404B. In the example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine the six coordinates of the handheld controller 400B relative to the wearable head-mounted device 400A (e.g., offsets in three translational directions and rotations along three axes). The six degrees of freedom can be represented relative to the coordinate system of the wearable head-mounted device 400A. The three translational offsets can be represented as X, Y, and Z offsets in such a coordinate system, as translation matrices, or as some other representation. The rotational degrees of freedom can be represented as a sequence of yaw, pitch, and roll rotations, as rotation matrices, as quaternions, or as some other representation. In some examples, a wearable head device 400A; one or more depth cameras 444 (and / or one or more non-depth cameras) included in the wearable head device 400A; and / or one or more optical targets (e.g., a button 350 of the handheld controller 400B as described above, or a dedicated optical target included in the handheld controller 400B) can be used for 6DOF tracking. In some examples, the handheld controller 400B may include a camera as described above; and the wearable head device 400A may include optical targets for combined camera optical tracking. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids for wirelessly transmitting and receiving three distinguishable signals. The 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined by measuring the relative amplitudes of the three distinguishable signals received in each of the coils used for receiving. In addition, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) that can be used to provide improved accuracy and / or more timely information about the rapid movement of the handheld controller 400B.

[0055] In some examples, it may become necessary to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head-mounted device 400A) to an inertial coordinate space (e.g., a coordinate space fixed relative to the real environment), for example, to compensate for movement of the wearable head-mounted device 400A relative to coordinate system 108. For example, such a transformation may be necessary for the display of the wearable head-mounted device 400A to present virtual objects at a desired position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of the wearable head-mounted device), rather than at a fixed position and orientation on the display (e.g., at the same position in the lower right corner of the display), to maintain the illusion that the virtual objects exist in the real environment (and, for example, not appear unnaturally positioned in the real environment when the wearable head-mounted device 400A moves and rotates). In some examples, the compensation transformation between coordinate spaces can be determined by processing images from depth camera 444 using SLAM and / or visual ranging procedures to determine the transformation of the wearable head-mounted device 400A relative to coordinate system 108. Figure 4 In the example shown, a depth camera 444 is coupled to a SLAM / visual ranging block 406 and can provide images to the block 406. SLAM / visual ranging block 406 implementations may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify a transformation between the head coordinate space and another coordinate space (e.g., inertial coordinate space). Similarly, in some examples, an additional source of information about the user's head pose and position is obtained from an IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual ranging block 406 to provide improved accuracy and / or more timely information regarding the user's head pose and position.

[0056] In some examples, depth camera 444 can feed 3D images to gesture tracker 411, which can be implemented in the processor of wearable head device 400A. Gesture tracker 411 can identify a user's gestures, for example, by matching the 3D image received from depth camera 444 with a stored pattern representing the gesture. Other suitable techniques for identifying user gestures will be apparent.

[0057] In some examples, one or more processors 416 may be configured to receive data from a 6DOF helmet subsystem 404B, IMU 409, SLAM / visual ranging block 406, depth camera 444, and / or gesture tracker 411 of a wearable head-mounted device. Processor 416 may also send and receive control signals from a 6DOF totem system 404A. Processor 416 may be wirelessly coupled to the 6DOF totem system 404A, as in an example where the handheld controller 400B is unrestricted. Processor 416 may also communicate with additional components such as audio-visual content memory 418, graphics processing unit (GPU) 420, and / or digital signal processor (DSP) audio spatializer 422. DSP audio spatializer 422 may be coupled to head-related transfer function (HRTF) memory 425. GPU 420 may include a left channel output coupled to a left source 424 of per-image modulated light and a right channel output coupled to a right source 426 of per-image modulated light. GPU 420 can output stereoscopic image data to image-by-image modulated light sources 424 and 426, as shown above relative to... Figures 2A-2D As described, the DSP audio sound field locator 422 can output audio to the left speaker 412 and / or the right speaker 414. The DSP audio sound field locator 422 can receive input from the processor 419 indicating a direction vector from the user to a virtual sound source (which can be moved by the user, e.g., via a handheld controller 300). Based on the direction vector, the DSP audio sound field locator 422 can determine the corresponding HRTF (e.g., by accessing the HRTF or by interpolating multiple HRTFs). The DSP audio sound field locator 422 can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This can enhance the credibility and realism of the virtual sound by interpolating the user's relative position and orientation to the virtual sound in the mixed reality environment—that is, by presenting a virtual sound that matches the user's expectation of what the virtual sound will sound like (if it is a real sound in a real environment).

[0058] In some examples, such as Figure 4 As shown, one or more of the processor 416, GPU 420, DSP audio sound field locator 422, HRTF memory 425, and audio / visual content memory 418 may be included in the auxiliary unit 400C (which may correspond to the auxiliary unit 320 described above). The auxiliary unit 400C may include a battery 427 that powers its components and / or powers the wearable head device 400A or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0059] Although Figure 4 Elements corresponding to various components of the example mixed reality system are presented, but various other suitable arrangements of these components will become apparent to those skilled in the art. For example, Figure 4 The components presented as associated with the auxiliary unit 400C may alternatively be associated with the wearable head device 400A or the handheld controller 400B. Furthermore, some mixed reality systems may completely omit the handheld controller 400B or the auxiliary unit 400C. Such changes and modifications will be understood to be included within the scope of the disclosed examples.

[0060] Ambient acoustic durability

[0061] As described above, an MRE (such as one experienced via a mixed reality system, e.g., mixed reality system 112, which may include components such as the wearable head unit 200, handheld controller 300, or assistive unit 320 described above) can present to the MRE user audio signals that appear to originate at a sound source having origin coordinates in the MRE. That is, the user can perceive these audio signals as if they were real audio signals originating from the origin coordinates of the sound source.

[0062] In some cases, audio signals can be considered virtual because they correspond to computational signals in a virtual environment. Virtual audio signals can be presented to the user as real audio signals that can be detected by the human ear, such as those generated via speakers 2134 and 2136 of the wearable head unit 200 in Figure 2.

[0063] The sound source can correspond to a real object and / or a virtual object. For example, a virtual object (e.g., Figure 1C The virtual monster (132) can emit audio signals in the MRE, which are represented as virtual audio signals in the MRE and presented to the user as real audio signals. For example, Figure 1C The virtual monster 132 can emit virtual sounds corresponding to the monster's voice (e.g., dialogue) or sound effects. Similarly, real objects (e.g., Figure 1CThe real object 122A appears to emit a virtual audio signal in the MRE, which is represented as a virtual audio signal in the MRE and presented to the user as a real audio signal. For example, a real light 122A can emit a virtual sound corresponding to the sound effect of the light being turned on or off—even if the light is not turned on or off in the real environment. The virtual sound can correspond to the location and orientation of a sound source (whether real or virtual). For example, if the virtual sound is presented to the user as a real audio signal (e.g., via speakers 2134 and 2136), the user can perceive the virtual sound as originating from the location of the sound source. The sound source is referred to herein as a "virtual sound source," even though the underlying object that obviously emits the sound can itself correspond to a real or virtual object, as described above.

[0064] Some virtual or mixed reality environments suffer from the problem of perceiving the environment as unreal or unbelievable. One reason for this perception is that audio and visual cues don't always match in such environments. For example, if a user is positioned behind a large brick wall in an MRE, the user might expect a sound coming from behind the wall to be quieter and more muffled than a sound originating next to the user. This expectation is based on the user's auditory experience in the real world, where sound becomes quiet and muffled as it passes through large, dense objects. When the user is presented with an audio signal that is supposedly originating behind the brick wall but is strong and presented at full volume, the illusion of the sound originating behind the brick wall is compromised. The entire virtual experience may feel fake and unreal, partly because it is not based on the user's expectation that real-world interactions fit into their expectations. Furthermore, in some cases, the "uncanny valley" problem arises, where subtle differences between virtual and real experiences can cause heightened feelings of discomfort. Improving the user experience by presenting audio signals in the MRE that appear to actually interact with objects in the user's environment—even in subtle ways—is desirable. Based on real-world experience, the more consistent the audio signal is with the user's expectations, the more immersive and engaging the user's MRE experience can be.

[0065] One way users perceive and understand their surroundings is through audio cues. In the real world, the actual audio signals a user hears are influenced by where those signals originate and what objects they interact with. For example, all other things being equal, a sound originating at a long distance from the user (e.g., a dog barking in the distance) will sound quieter than the same sound originating at a short distance (e.g., a dog barking in the same room as the user). Therefore, a user can partially identify the dog's location in the real environment based on the perceived volume of the bark. Similarly, all other things being equal, a sound moving away from the user (e.g., the voice of a person with their back to the user) will sound less distinct and less audible (i.e., low-pass filtered) than the same sound moving towards the user (e.g., the voice of a person facing the user). Therefore, a user can identify a person's orientation in the real environment based on the perceived characteristics of that person's voice.

[0066] The user's perception of a real audio signal can also be influenced by the presence of objects in the environment that interact with the audio signal. That is, the user can perceive not only the audio signal generated by the sound source, but also its reflection from nearby objects and the reverberation characteristics imparted by the surrounding acoustic space. For example, if a user speaks in a small room with closed walls, those walls may produce a short, natural reverberation signal because the speaker's voice is reflected away by the walls. The user can infer from that reverberation that they are in a small room with closed walls. Similarly, a large concert hall or church may produce a longer reverberation, from which the user can infer that they are in a large, spacious room. Likewise, the reverberation of an audio signal can present various acoustic characteristics based on the location or orientation of the surfaces from which the signal reflects, or the materials of those surfaces. For example, reverberation against a tiled wall will sound different from reverberation against brick, carpet, drywall, or other materials. These reverberation characteristics can be used by the user to understand—acoustically—the size, shape, and material composition of the space they inhabit.

[0067] The examples above illustrate how audio cues can inform a user's perception of their surroundings. These cues can work in conjunction with visual cues: for example, if we see a dog in the distance, a user can expect the sound of that dog barking to correspond to that distance (and if it doesn't, the user may feel confused or disoriented, as in some virtual environments). In some examples, such as in low-light environments, or for visually impaired users, visual cues may be limited or unavailable; in such cases, audio cues can take on specific importance and can serve as the primary means for users to understand their environment.

[0068] The system architecture may facilitate the organization, storage, retrieval, and / or management of information required to render realistic virtual audio. For example, an MR system (e.g., MR systems 112, 200) can manage environmental information, such as the real-world environment in which a user might be, the acoustic characteristics of that environment, and / or the user's location within that environment. MR systems can further manage information about objects in the real and / or virtual environments (e.g., objects that may affect the general acoustic characteristics of the real environment and / or objects that may affect the acoustic characteristics of virtual sound sources interacting with those objects). MR systems can also manage information about virtual sound sources. For example, the location of virtual sound sources may be relevant to rendering realistic virtual audio.

[0069] In addition to managing the virtual audio system, other systems may also need to be managed to deliver a complete MR experience. For example, a complete MR experience might require a virtual vision system that manages information used to render virtual objects. A complete MR experience might also require a simultaneous localization and mapping ("SLAM") system that builds, updates, and / or maintains a 3D model of the user's environment. Besides the virtual audio system, MR systems (e.g., MR systems 112, 200) can manage these and more systems to deliver a complete MR experience. The virtual audio system architecture may facilitate the management of interactions between these systems to promote data transfer, management, storage, and / or security.

[0070] In some embodiments, a system (e.g., a virtual audio system) can interact with other higher-level systems. In some embodiments, a lower-level system (e.g., a virtual audio system) can interact more closely with hardware-level inputs and / or outputs, while a higher-level system (e.g., an application) can interact with a lower-level system. Higher-level systems can utilize lower-level systems to perform their functions (e.g., a gaming application might rely on a lower-level virtual audio system to render realistic virtual audio). Virtual audio systems can benefit from system architectures designed to manage interactions with higher-level systems while maintaining the integrity of the virtual audio system. For example, multiple higher-level systems (e.g., multiple third-party applications) can interface with a virtual audio system simultaneously or substantially simultaneously. In some embodiments, maintaining a single virtual audio system capable of rendering virtual audio may be computationally more efficient than having each higher-level system maintain a separate audio system. For example, in some embodiments, a single digital reverb can be used to process sound objects from multiple higher-level systems when those objects are intended to be located in the same virtual or real acoustic space (e.g., the user's room). A well-designed system architecture can also protect the integrity of information that might be used in other applications (e.g., preventing data corruption and / or tampering).

[0071] In some embodiments, it may be advantageous to design a system architecture that allows for real-time changes without interrupting services to other systems (e.g., higher-level systems). For example, a virtual audio system may store, maintain, or otherwise manage an audio model that takes into account the acoustic properties of a real-world environment (e.g., a room). If a user changes the real-world environment (e.g., moves to a different room), the audio model may be updated to account for the change. If an MR system is currently in use (e.g., the MR system is presenting virtual visuals and / or virtual audio to a user), the audio model may need to be updated while other systems (e.g., higher-level systems) are still using the audio model to render virtual audio.

[0072] In some embodiments, it may be advantageous to design a system architecture that allows changes to be propagated to other systems. For example, some systems may maintain a separate copy of the audio model, or some systems may store specific repetitive sound effects rendered using an audio model maintained by a virtual audio system. Therefore, it may be advantageous to propagate changes made in the virtual audio system to other systems. For example, if a user changes the environment (e.g., moves the room) and a new audio model may be more accurate, the virtual audio system can modify its audio model and notify any clients (e.g., systems that use and / or depend on the virtual audio system) of the change. In some embodiments, the client can then query the virtual audio system and update its internal data accordingly.

[0073] Figure 5An exemplary virtual audio system according to some embodiments is illustrated. The virtual audio system 500 may include a persistent module 502. A module (e.g., persistent module 502) may include one or more computer systems configured to execute instructions and / or store one or more data structures. In some embodiments, the module (e.g., persistent module 502) may be configured to execute processes, subprocesses, threads, and / or services managed by an audio service 522 (e.g., instructions executed by persistent module 502 may run within audio service 522), which may run on one or more computer systems. In some embodiments, audio service 522 may be a process that can run in a runtime environment, and instructions executed by the module (e.g., persistent module 502) may be components of audio service 522 (e.g., instructions executed by persistent module 502 may be child processes of audio service 522). In some embodiments, audio service 522 may be a child process of a parent process. Instructions executed by a module (e.g., persistent module 502) may include one or more components (e.g., processes, subprocesses, threads, and / or services executed by the positioning state submodule 506, acoustic data submodule 508, and / or audio model submodule 510). In some embodiments, instructions executed by a module (e.g., persistent module 502) may run as a subprocess of audio service 522 and / or as a separate process in a location different from other components of audio service 522. For example, instructions executed by a module (e.g., persistent module 502) may run in a general-purpose processor, and one or more other components of audio service 522 may run in an audio-specific processor (e.g., DSP). In some embodiments, instructions executed by a module (e.g., persistent module 502) may run in a process address space and / or memory space different from other components of audio service 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may run as one or more threads within audio service 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may be instantiated within the audio service 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may share process addresses and / or memory space with other components of the audio service 522.

[0074] In some embodiments, the persistence module 502 may include a location state submodule 506. The location state submodule 506 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the location state submodule 506 may be a subprocess of the persistence module 502. In some embodiments, the location state submodule 506 may indicate whether location has been achieved (e.g., the location state submodule 506 may indicate whether the MR system has identified the real environment and / or located itself in the real environment). In some embodiments, the location state submodule 506 may interface with a location system (e.g., via an API). The location system may determine the location for the MR system (and / or the user using the MR system). In some embodiments, the location system may utilize SLAM-like techniques to create a 3D model of the real environment and estimate the system's (and / or the user's) location within the environment. In some embodiments, the location system may rely on a traversable world system (described further in detail below) and (e.g., one or more sensors of MR systems 112, 200) to estimate the MR system's (and / or the user's) location within the environment. In some embodiments, the positioning status submodule 506 may query the positioning system to determine whether positioning for the MR system has been achieved. Similarly, the positioning system may notify the positioning status submodule 506 of successful positioning. (For example, the positioning status of MR systems 112, 200) can be used to determine whether the audio model should be updated (e.g., because the user's real-world environment has changed).

[0075] In some embodiments, the persistence module 502 may include an acoustic data submodule 508. The acoustic data submodule 508 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the acoustic data submodule 508 may be a subprocess of the persistence module 502. In some embodiments, the acoustic data submodule 508 may store one or more data structures representing acoustic data that can be used to create an audio model. In some embodiments, the acoustic data submodule 508 may interface with a walkable world system (e.g., via an API). The walkable world system may include information about a known real-world environment (e.g., rooms, buildings, and / or external space) and associated real and / or virtual objects. In some embodiments, the walkable world system may include persistent coordinate frames and / or anchor points. Persistent coordinate frames and / or anchor points may be fixed points in a space that the MR system may know (e.g., by unique identifiers). Virtual objects can be positioned relative to one or more persistent coordinate frames and / or anchor points to enable object persistence (e.g., a virtual object can appear to remain in the same location in the real environment regardless of who is viewing it and regardless of any user movement). Persistent coordinate frames and / or anchor points can be particularly advantageous when two or more users with separate MR systems use different world coordinate systems (e.g., each user's location is specified as the origin for their respective world coordinate system). Object persistence across users can be achieved by converting between individual world coordinate systems and a common persistent coordinate frame, and by placing / referencing virtual objects relative to persistent coordinate frames. In some embodiments, a walkable world system can manage and maintain persistent coordinate frames by, for example, mapping new regions and creating new persistent coordinate frames; remapping known regions and harmonizing new persistent coordinate frames with previously determined persistent coordinate frames; and / or associating persistent coordinate frames with identifiable information (e.g., location and / or nearby objects). In some embodiments, the acoustic data submodule 508 can query one or more persistent coordinate frames to a separate system (e.g., a walkable world system). In some embodiments, the acoustic data submodule 508 may retrieve associated persistent coordinate frames (e.g., persistent coordinate frames within a threshold radius of the user's location) to facilitate access and management, including the creation, modification, and / or deletion of associated acoustic data.

[0076] In some embodiments, acoustic data stored in the acoustic data submodule 508 may be organized into physically relevant modular units (e.g., a room may be represented by a modular unit, and a chair within the room may be represented by another modular unit). For example, a modular unit may include physical and / or perceived properties of the physical environment (e.g., the room). Physical and / or perceived properties may include attributes that may affect the acoustic characteristics of the room (e.g., the room's size and / or shape). In some embodiments, physical and / or perceived properties may include functional and / or behavioral attributes that can be interpreted by a rendering engine (e.g., whether sources outside the room should be occluded). In some embodiments, physical and / or perceived properties may include attributes of known and / or identified objects. For example, the geometry of fixed (e.g., floors, walls, furniture, etc.) and / or movable (e.g., cups) objects may be stored as physical and / or perceived properties and may be associated with a specific environment. In some embodiments, physical and / or perceived properties may include transmission loss, scattering coefficient, and / or absorption coefficient. In some embodiments, a modular unit may include (e.g., between other modular units or within a modular unit) physically and / or perceived links. For example, physical and / or sensory-related links can link two or more rooms together and describe how the rooms can interact with each other (e.g., cross-coupling gain levels between digital reverberators simulating the rooms and / or line-of-sight paths between two spaces).

[0077] In some embodiments, physical and / or perceived properties may include acoustic properties such as reverberation time, reverberation delay, and / or reverberation gain. Reverberation time may include the length of time required for sound to decay by a certain amount (e.g., 60 dB). Sound decay may be a result of sound reflecting off surfaces in the real environment (e.g., walls, floors, furniture, etc.) while energy is lost due to sound absorption by, for example, room boundaries (e.g., walls, floors, ceilings, etc.), objects within the room (e.g., chairs, furniture, people, etc.), and the air within the room. Reverberation time may be affected by environmental factors. For example, absorbent surfaces (e.g., cushions) can absorb sound in addition to geometric expansion and thus reduce reverberation time. In some embodiments, information about the original source may not be required to estimate the reverberation time of the environment. Reverberation gain may include the ratio of the direct / source / original energy of the sound to the reverberant energy of the sound (e.g., the energy of the reverberation produced by the direct / source / original sound), where the listener and source are substantially co-located (e.g., a user clapping their hands produces a source sound that can be considered substantially co-located with one or more microphones mounted on a head-mounted MR system). For example, a pulse (e.g., clapping) can have energy associated with that pulse, and the reverberant sound from that pulse can have energy associated with the reverberation of that pulse. The ratio of the original / source energy to the reverberant energy can be the reverberation gain. The reverberation gain in a real environment may be affected by, for example, absorbing surfaces that can absorb sound and thus reduce reverberation energy.

[0078] In some embodiments, acoustic data may include metadata (e.g., metadata about physical and / or perceived properties). For example, information about when and / or where acoustic data was collected may be included in the acoustic data. In some embodiments, confidence data associated with the acoustic data (e.g., estimated measurement accuracy and / or counts of repeated measurements) may be included as metadata. In some embodiments, a type of modular unit (e.g., a modular unit for a room or a link between modular units) and / or a data version may be included as metadata. In some embodiments, unique identifiers associated with the acoustic data, persistent coordinate frames, and / or anchor points may be included as metadata. In some embodiments, relative transformations from persistent coordinate frames and / or anchor points and associated virtual objects may be included as metadata. In some embodiments, metadata may be stored together with the acoustic data as a single bundle.

[0079] In some embodiments, acoustic data may be organized using persistent coordinate frames and / or anchor points, and these persistent coordinate frames and / or anchor points may be organized into a map. In some embodiments, an audio model may take into account acoustic data organized using persistent coordinate frames and / or anchor points, which may correspond to locations within the environment. In some embodiments, acoustic data may be loaded into the acoustic data submodule 508 upon a successful localization event (which may be indicated by the localization status submodule 506). In some embodiments, all available acoustic data may be loaded into the acoustic data submodule 508. In some embodiments, only relevant acoustic data may be loaded into the acoustic data submodule 508 (e.g., acoustic data for persistent coordinate frames and / or anchor points within a specific distance of the MR system's location).

[0080] In some embodiments, acoustic data may include different states that can change according to variations in the real environment. For example, acoustic data representing a given modular unit of a room may include acoustic data of a room in an empty state and acoustic data of a room in an occupied state. In some embodiments, changes in the arrangement of furniture in a room may be reflected by changes in the state of acoustic data associated with the room. In some embodiments, a modular unit (e.g., representing a room) may include different acoustic data for the state when a door is open or closed. The state may be represented as a binary value (e.g., 0 or 1) or a continuous value (e.g., the degree of door opening, the occupancy of the room, etc.).

[0081] In some embodiments, persistence module 502 may include audio model submodule 510. Audio model submodule 510 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, audio model submodule 510 may include one or more data structures representing audio models for real and / or virtual environments. In some embodiments, the audio model may be generated at least in part from acoustic data stored in acoustic data submodule 508. The audio model may represent how sound behaves in a particular environment. For example, the audio model in audio model submodule 510 may modify virtual sounds generated by an MR system (e.g., MR system 112, 200) to reflect the acoustic properties of the environment. A virtual concert presented to a user sitting in a large, empty concert hall may have similar acoustic properties to a real concert presented in the same concert hall. The MR system may locate itself to the identified concert hall, load relevant acoustic data, and generate an audio model to model the acoustic properties of the concert hall.

[0082] In some embodiments, the audio model can be used to model sound propagation in an environment. For example, propagation effects may include occlusion, blockage, early reflection, diffraction, time-of-flight delay, Doppler effect, and other effects. In some embodiments, the audio model may take into account frequency-related absorption and / or transmission losses (e.g., based on acoustic data loaded into the acoustic data submodule 508). In some embodiments, the audio model stored in the audio model submodule 510 may inform other aspects of the audio engine. For example, the audio model may use acoustic data to procedurally synthesize audio (e.g., collisions between virtual and / or real objects).

[0083] In some embodiments, the audio rendering service 522 may include a rendering track module 514. The rendering track module 514 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the rendering track module 514 may include audio information that can later be presented to a user. In some embodiments, the MR system may present virtual sounds comprising several sound sources mixed together (e.g., the sound of two swords clashing and the sound of a person shouting). The rendering track module 514 may store one or more tracks that can be mixed with other tracks to be presented to a user. In some embodiments, the rendering track module 514 may include information about spatial sources. For example, the rendering track module 514 may include information about where the sound sources are located, which may be taken into account in the audio model and / or rendering algorithm. In some embodiments, the rendering track module 514 may include information about the relationships between modular units and / or sound sources. For example, one or more rendering tracks and / or audio models may be associated together as a single group.

[0084] In some embodiments, the audio rendering service 522 may include a position manager module 516. The position manager module 516 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the position manager module 516 may manage location information associated with the audio engine (e.g., the current location of the MR system in a real-world environment). In some embodiments, the position manager module 516 may include a perception wrapper submodule. The perception wrapper submodule may be a wrapper around perception data (e.g., what the MR system has detected or is detecting). In some embodiments, the perception wrapper may interface with and / or convert between the perception data and the position manager module 516. In some embodiments, the position manager module 516 may include a head pose submodule, which may include head pose data. The head pose data may include the location and / or orientation of the MR system (or the corresponding user) in a real-world environment. In some embodiments, the head pose may be determined based on the perception data.

[0085] In some embodiments, audio rendering service 522 may include audio model module 518. Audio model module 518 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, audio model module 518 may include an audio model, which may be the same audio model included in audio model submodule 510. In some embodiments, modules 510 and 518 may maintain duplicate copies of the same audio model. For example, maintaining more than one copy of the audio model may be advantageous when updating the audio model but needing to present sound to the user. It may be advantageous to update the copy of the model when the model is currently in use and then update the outdated model when it becomes available (e.g., no longer in use). In some embodiments, the audio model may be transferred between modules 510 and 518 via serialization. The audio model in module 510 may be serialized and deserialized to facilitate data transfer to module 518. Serialization may facilitate data transfer between processors (e.g., general-purpose processors and audio-specific processors) so that shared typed memory is not required.

[0086] In some embodiments, the audio rendering service 522 may include a rendering algorithm module 520. The rendering algorithm module 520 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the rendering algorithm module 520 may include algorithms for rendering virtual sounds so that they can be presented to a user (e.g., through one or more speakers of an MR system). The rendering algorithm module 520 may take into account an audio model specific to an environment (e.g., the audio model in modules 510 and / or 518).

[0087] In some embodiments, audio service 522 may be a process, subprocess, thread, and / or service running on one or more computer systems (e.g., in MR systems 112, 200). In some embodiments, a separate system (e.g., a third-party application) may request the presentation of audio signals (e.g., via one or more speakers of MR systems 112, 200). Such requests may take any suitable form. In some embodiments, a request to present audio signals may include software instructions to present audio signals; in some embodiments, such a request may be hardware-driven. Requests may be made with or without user involvement. Further, such requests may be received via local hardware (e.g., from the MR system itself), via external hardware (e.g., a separate computer system communicating with the MR system), via the Internet (e.g., via a cloud server), or via any other suitable source or combination of sources. In some embodiments, audio service 522 may receive the request, render the requested audio signal (e.g., via rendering algorithm 520, which may take into account the audio models from blocks 510 and / or 518), and present the requested audio signal to the user. In some embodiments, audio service 522 may be a process that runs continuously (e.g., in the background) while the operating system of the MR system is running. In some embodiments, audio service 522 may be an instantiation of a parent background service, which may serve as the main process of one or more background processes and / or child processes. In some embodiments, audio service 522 may be part of the operating system of the MR system. In some embodiments, applications that can run on the MR system can access audio service 522. In some embodiments, the user of the MR system may not provide input directly to audio service 522. For example, the user may provide input (e.g., movement commands) to an application running on the MR system (e.g., a role-playing game). The application may provide input to audio service 522 (e.g., to render footsteps), and audio service 522 may provide output (e.g., rendered footsteps) to the user (e.g., via a speaker) and / or other processes and / or services.

[0088] Figure 6 An exemplary process for updating an audio model according to some embodiments is shown. At step 606, a location may be determined (e.g., the MR system can successfully identify its location within the environment). At step 607, which may occur within persistence module 602 (which may correspond to persistence module 502), a notification of successful location may be issued. In some embodiments, the notification of successful location may trigger the process of updating the audio model (e.g., because the previous audio model may no longer be applicable to the current location).

[0089] At step 608, it can be determined whether to initiate a call to the acoustic data. It may be desirable to set one or more conditions for initiating the call so that the audio model is not updated too frequently. For example, if a user using the MR system only moves slightly within the room, updating the audio model may not be desirable (e.g., because an updated model may not be perceptually distinguishable from the existing model, and / or continuous model updates may be computationally expensive). In some embodiments, the threshold condition at step 608 may be time-based. For example, a call may be initiated only if it has not been initiated within the previous 5 seconds. In some embodiments, the threshold condition at step 608 may be location-based. For example, a call may be initiated only if the user has changed their location by a threshold distance. It should be noted that other threshold conditions may also be used. In some embodiments, steps 607 and / or 608 may occur within a location state submodule (e.g., location state submodule 506).

[0090] If it is determined that a call should be initiated, persistent coordinate frames can be retrieved at step 610. In some embodiments, only a subset of available persistent coordinate frames can be retrieved at step 610. For example, only persistent coordinate frames near the location can be retrieved.

[0091] At step 612, acoustic data may be retrieved. In some embodiments, the acoustic data retrieved at step 612 may correspond to acoustic data stored in acoustic data submodule 508. In some embodiments, only a subset of the available acoustic data may be retrieved at step 612. For example, acoustic data associated with one or more persistent coordinate frames and / or anchor points may be retrieved. In some embodiments, steps 612 and / or 614 may occur within the acoustic data submodule (e.g., acoustic data submodule 508).

[0092] At step 614, an audio model may be constructed and / or modified. In some embodiments, the audio model may take into account the acoustic data retrieved at step 612, and the audio model may model the acoustic characteristics of a particular environment. In some embodiments, step 614 may occur within an audio model submodule (e.g., audio model submodule 510).

[0093] At step 616, it can be determined whether a copy of the audio model should be updated. It may be desirable to set one or more conditions for updating the copy of the audio model to avoid service interruption (e.g., presenting audio to the user). For example, a condition may assess whether a copy of the audio model exists (e.g., within audio rendering service 604 but outside of persistence module 602). If a copy of the audio model does not exist, the audio model can be retrieved by audio rendering service 604 (which may correspond to audio rendering service 522). In some embodiments, a condition may assess whether a copy of the audio model is currently in use (e.g., whether the audio model is being used to render audio for presentation to the user). If a copy of the audio model is not in use, an updated audio model (e.g., the audio model generated at step 614) can be propagated to that copy.

[0094] At step 618, the audio rendering service 604 may retrieve a copy of the audio model (e.g., the audio model generated in step 614). The audio model may be transmitted using data serialization and / or deserialization.

[0095] At step 620, outdated audio models may optionally be deleted and / or disabled. For example, audio rendering service 604 may have a first existing audio model that it has previously used. Audio rendering service 604 may retrieve a second updated audio model (e.g., from persistence module 602) and delete and / or disable the first existing audio model.

[0096] At step 622, a notification of the new audio model may be issued. In some embodiments, the notification may include a callback function for clients (e.g., third-party applications) that can subscribe to hear the audio model when it changes.

[0097] Figure 7 An exemplary process for updating an audio model according to some embodiments is illustrated. At step 706, audio data may be received (e.g., via one or more sensors of MR systems 112, 200). In some embodiments, audio data may be manually entered (e.g., a user and / or developer may manually enter reverberation time, reverberation delay, reverberation gain, etc.). At step 708, which may occur in persistence module 702 (which may correspond to persistence module 502), the associated environment may be identified. The associated environment may be identified by metadata that may accompany the audio data (e.g., the metadata may carry information about anchor points and / or one or more persistent coordinate frames that may be known to the MR system).

[0098] At step 710, it can be determined whether the associated environment is new. For example, if the associated environment may not be identified and / or associated with an unknown identifier, it can be determined that the audio data is associated with a new environment. If it is determined that the associated environment is not new, the copy of the official audio model can be updated (e.g., utilizing room properties that can be derived from the audio data). If it is determined that the associated environment is new, the new environment can be added to the copy of the official audio model, and the copy audio model can be updated accordingly. In some embodiments, the new environment may be represented by a new modular unit.

[0099] At step 716, metadata associated with the new environment (e.g., metadata associated with the new modular unit) can be initialized. For example, metadata associated with measurement counts, confidence levels, or other information can be created and bundled with the new modular unit.

[0100] At step 718, the official audio model within the persistence module 702 can be updated. For example, the official audio model can be copied from an updated copy audio model. In some embodiments, the official audio model can be locked at step 718 to prevent further changes to the official audio model. In some embodiments, changes can still be made to a copy of the official audio model (which may still exist within the persistence module 702) even when the official audio model is locked.

[0101] At step 720, acoustic data associated with the new audio data may be saved. For example, a new modular unit associated with a new room may be saved and / or transferred to a passable world system (which can then be accessed by the MR system as needed in the future). In some embodiments, step 720 may occur within the acoustic data submodule 508. In some embodiments, step 720 may occur sequentially after step 718. In some embodiments, step 720 may occur at an independent time, as determined by other components, for example, based on the availability of a passable world system where acoustic data can be saved.

[0102] At step 722, audio rendering service 704 (which may correspond to audio rendering service 522) may retrieve a copy of the audio model. In some embodiments, the copy of the audio model may be the same audio model updated at step 718. In some embodiments, data transfer may occur via serialization and deserialization. In some embodiments, the audio model may be locked during the serialization process, which may prevent changes to the audio model when a snapshot of the audio model is created.

[0103] At step 724, outdated copies of the audio model can be deleted and / or disabled.

[0104] At step 726, a notification about the new audio model can be issued. This notification can be a callback function for subscribed clients to hear when the model is updated.

[0105] At step 728, the official audio model can be released, which may indicate that the serialization package corresponding to the audio model can be deleted. In some embodiments, it may be desirable to lock the official audio model within the persistence module 702 when a copy of the official audio model is sent to the audio rendering service 704. Once the audio rendering service 704 has finished retrieving the copy of the official audio model, it may be desirable to release the lock on the official audio model so that the official audio model can continue to be updated.

[0106] In some embodiments, the audio rendering service (e.g., audio rendering service 522) may not manage the interaction between the persistence module (e.g., persistence module 502) and the rendering algorithm (e.g., rendering algorithm module 520). For example, the rendering algorithm 520 may communicate directly with the persistence module 502 to retrieve updated audio models. In some embodiments, the rendering algorithm 520 may include its own copy of the audio model. In some embodiments, the rendering algorithm 520 may access the audio model within the persistence module 502.

[0107] As provided above, this paper discloses systems and methods for storing, organizing, and maintaining acoustic data for mixed reality systems.

[0108] A system may include one or more sensors of a head-mounted device, a speaker of the head-mounted device, and one or more processors configured to perform a method. The method, executed by the one or more processors, may include: receiving a request to present an audio signal; identifying an environment via one or more sensors of the head-mounted device; retrieving one or more audio model components associated with the environment; generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and the request to present the audio signal; and presenting the modified audio signal via the speaker of the head-mounted device.

[0109] In some system aspects, the second audio model can be generated by an audio service. In some system aspects, the modified audio signal can be determined by an audio service. In some system aspects, the second audio model can be a copy of the first audio model.

[0110] In some system aspects, one or more audio model components may include one or more dimensions of the environment. In some system aspects, one or more audio model components may include reverberation time. In some system aspects, one or more audio model components may include reverberation gain. In some system aspects, one or more audio model components may include transmission loss coefficients. In some system aspects, one or more audio model components may include absorption coefficients.

[0111] A method may include receiving a request to present an audio signal. The environment may be identified via one or more sensors of a head-mounted device. One or more audio model components associated with the environment may be retrieved. A first audio model based on the audio model components may be generated. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and the request to present the audio signal. The modified audio signal may be presented via a speaker of the head-mounted device.

[0112] In some aspects of the method, the second audio model may be generated by an audio service. In some aspects of the method, the modified audio signal may be determined by an audio service. In some aspects of the method, the second audio model may be a copy of the first audio model.

[0113] In some aspects of the method, one or more audio model components may include one or more dimensions of the environment. In some aspects of the method, one or more audio model components may include reverberation time. In some aspects of the method, one or more audio model components may include reverberation gain. In some aspects of the method, one or more audio model components may include transmission loss coefficients. In some aspects of the method, one or more audio model components may include absorption coefficients.

[0114] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method, to be executed by one or more processors, may include: receiving a request to present an audio signal; identifying an environment via one or more sensors of a head-mounted device; retrieving one or more audio model components associated with the environment; generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and the request to present the audio signal; and presenting the modified audio signal via a speaker of the head-mounted device.

[0115] In some non-transitory computer-readable media, the second audio model may be generated by an audio service. In some non-transitory computer-readable media, the modified audio signal may be determined by an audio service. In some non-transitory computer-readable media, the second audio model may be a copy of the first audio model.

[0116] In some non-transitory computer-readable media, one or more audio model components may include one or more dimensions of the environment. In some non-transitory computer-readable media, one or more audio model components may include reverberation time. In some non-transitory computer-readable media, one or more audio model components may include reverberation gain. In some non-transitory computer-readable media, one or more audio model components may include transmission loss coefficients. In some non-transitory computer-readable media, one or more audio model components may include absorption coefficients.

[0117] A system may include one or more sensors of a head-mounted device, a speaker of the head-mounted device, and one or more processors configured to perform a method. The method, executed by the one or more processors, may include: receiving audio data via one or more sensors of the head-mounted device; determining one or more acoustic characteristics of an environment based on the audio data; determining the associated environment for the audio data; generating an updated audio model based on the one or more acoustic characteristics of the environment; retrieving the updated audio model; generating a notification associated with the updated audio model; and presenting an audio signal based on the updated audio model via the speaker of the head-mounted device.

[0118] In some system aspects, updated audio models can be retrieved by the audio service. In some system aspects, notifications can be generated by the audio service. In some system aspects, generating updated audio models can be further based on previous audio models.

[0119] In some system aspects, one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some system aspects, one or more acoustic characteristics of the environment may include reverberation time. In some system aspects, one or more acoustic characteristics of the environment may include reverberation gain. In some system aspects, one or more acoustic characteristics of the environment may include transmission loss coefficient. In some system aspects, one or more acoustic characteristics of the environment may include absorption coefficient.

[0120] A method may include receiving audio data via one or more sensors of a head-mounted device. One or more acoustic characteristics of an environment may be determined based on the audio data. An associated environment may be determined for the audio data. An updated audio model may be generated based on one or more acoustic characteristics of the environment. The updated audio model may be retrieved. A notification associated with the updated audio model may be generated. An audio signal based on the updated audio model may be presented via a speaker of the head-mounted device.

[0121] In some methodologies, the updated audio model can be retrieved by the audio service. In some methodologies, the notification can be generated by the audio service. In some methodologies, generating the updated audio model can be further based on a previous audio model.

[0122] In some aspects of the method, one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some aspects of the method, one or more acoustic characteristics of the environment may include reverberation time. In some aspects of the method, one or more acoustic characteristics of the environment may include reverberation gain. In some aspects of the method, one or more acoustic characteristics of the environment may include transmission loss coefficient. In some aspects of the method, one or more acoustic characteristics of the environment may include absorption coefficient.

[0123] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method, to be executed by one or more processors, may include: receiving audio data via one or more sensors of a head-mounted device; determining one or more acoustic properties of an environment based on the audio data; determining an environment associated with the audio data; generating an updated audio model based on the one or more acoustic properties of the environment; retrieving the updated audio model; generating a notification associated with the updated audio model; and presenting an audio signal based on the updated audio model via a speaker of the head-mounted device.

[0124] In some non-transitory computer-readable media, updated audio models can be retrieved by the audio service. In some non-transitory computer-readable media, notifications can be generated by the audio service. In some non-transitory computer-readable media, generating updated audio models can be further based on previous audio models.

[0125] In some non-transitory computer-readable media, one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some non-transitory computer-readable media, one or more acoustic characteristics of the environment may include reverberation time. In some non-transitory computer-readable media, one or more acoustic characteristics of the environment may include reverberation gain. In some non-transitory computer-readable media, one or more acoustic characteristics of the environment may include transmission loss coefficient. In some non-transitory computer-readable media, one or more acoustic characteristics of the environment may include absorption coefficient.

[0126] While the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more embodiments may be combined, deleted, modified, or supplemented to form further embodiments. Such changes and modifications will be understood to be included within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A system comprising: One or more sensors in a head-mounted device; The speaker of the head-mounted device; One or more processors are configured to perform a method comprising: The environment of the head-mounted device is identified via one or more sensors of the head-mounted device; At the first moment: The user's first position relative to the mixed reality environment is detected via the one or more sensors; Retrieve one or more audio model components associated with the environment, the one or more audio model components including anchor points associated with the environment, wherein the anchor points include fixed points in the environment known to the system; and Based on the one or more audio model components, a first audio model is generated, wherein the first audio model is associated with the mixed reality environment; At a second time later than the first time: Receive requests to present audio signals; The second position of the user of the head-mounted device relative to the mixed reality environment is detected via the one or more sensors; Determine whether the difference between the first position and the second position exceeds a threshold; Based on the determination that the difference exceeds the threshold: Based on the first audio model, a second audio model is generated; and Based on the second audio model, the audio signal is determined; and Based on the determination that the difference does not exceed the threshold: Based on the first audio model, the audio signal is determined; and The audio signal is presented via the speaker of the headset.

2. The system according to claim 1, wherein, The second audio model is generated via an audio service.

3. The system according to claim 1, wherein, The audio signal is determined via an audio service.

4. The system according to claim 1, wherein, The one or more audio model components include one or more dimensions of the environment.

5. The system according to claim 1, wherein, The one or more audio model components include reverberation time.

6. The system according to claim 1, wherein, The one or more audio model components include reverb gain.

7. The system according to claim 1, wherein, The one or more audio model components include transmission loss coefficients.

8. The system according to claim 1, wherein, The one or more audio model components include absorption coefficients.

9. A method comprising: The environment of the head-mounted device is identified via one or more sensors of the head-mounted device; At the first moment: The user's first position relative to the mixed reality environment is detected via the one or more sensors; Retrieve one or more audio model components associated with the environment, the one or more audio model components including anchor points associated with the environment, wherein the anchor points include fixed points in the environment known to the system; and Based on the one or more audio model components, a first audio model is generated, wherein the first audio model is associated with the mixed reality environment; At a second time later than the first time: Receive requests to present audio signals; The second position of the user of the head-mounted device relative to the mixed reality environment is detected via the one or more sensors; Determine whether the difference between the first position and the second position exceeds a threshold; Based on the determination that the difference exceeds the threshold: Based on the first audio model, a second audio model is generated; and Based on the second audio model, the audio signal is determined; and Based on the determination that the difference does not exceed the threshold: Based on the first audio model, the audio signal is determined; and The audio signal is presented via the speaker of the headset.

10. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: The environment of the head-mounted device is identified via one or more sensors of the head-mounted device; At the first moment: The user's first position relative to the mixed reality environment is detected via the one or more sensors; Retrieve one or more audio model components associated with the environment, the one or more audio model components including anchor points associated with the environment, wherein the anchor points include fixed points in the environment known to the system; and Based on the one or more audio model components, a first audio model is generated, wherein the first audio model is associated with the mixed reality environment; At a second time later than the first time: Receive requests to present audio signals; The second position of the user of the head-mounted device relative to the mixed reality environment is detected via the one or more sensors; Determine whether the difference between the first position and the second position exceeds a threshold; Based on the determination that the difference exceeds the threshold: Based on the first audio model, a second audio model is generated; and Based on the second audio model, the audio signal is determined; and Based on the determination that the difference does not exceed the threshold: Based on the first audio model, the audio signal is determined; and The audio signal is presented via the speaker of the headset.

11. The non-transitory computer-readable medium according to claim 10, wherein, The second audio model is generated via an audio service.

12. The non-transitory computer-readable medium according to claim 10, wherein, The audio signal is determined via an audio service.

13. The non-transitory computer-readable medium according to claim 10, wherein, The one or more audio model components include one or more dimensions of the environment.

14. The non-transitory computer-readable medium according to claim 10, wherein, The one or more audio model components include reverberation time.

15. The non-transitory computer-readable medium according to claim 10, wherein, The one or more audio model components include reverb gain.

16. The non-transitory computer-readable medium according to claim 10, wherein, The one or more audio model components include transmission loss coefficients.

17. The non-transitory computer-readable medium according to claim 10, wherein, The one or more audio model components include absorption coefficients.

Images