Environmental Sound Persistence

Abstract

To provide appropriate environment acoustics persistence.SOLUTION: Disclosed herein are systems and methods for storing, organizing, and maintaining acoustic data for mixed reality systems. A system may include one or more sensors of a head-wearable device, a speaker of the head-wearable device, and one or more processors configured to execute a method. A method for execution by the one or more processors may include receiving a request to present an audio signal. An environment may be identified via the one or more sensors of the head-wearable device. One or more audio model components associated with the environment may be retrieved.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 944,956, filed December 6, 2019, the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to systems and methods for managing and storing audio data, and more particularly to systems and methods for managing and storing audio data within a mixed reality environment. [Background technology]

[0003] Virtual environments are ubiquitous in computing environments, finding use in video games (where a virtual environment may represent a game world), maps (where a virtual environment may represent a terrain to be navigated), simulations (where a virtual environment may simulate a real environment), digital storytelling (where virtual characters may interact with one another within a virtual environment), and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, a user's experience with a virtual environment may be limited by the technology for presenting the virtual environment. For example, conventional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may be incapable of realizing a virtual environment in a way that creates a compelling, realistic, and immersive experience.

[0004] Virtual reality ("VR"), augmented reality ("AR"), mixed reality ("MR"), and related technologies (collectively, "XR") share the ability to present to a user of an XR system sensory information corresponding to a virtual environment represented by data in a computer system. Such systems can provide a uniquely increased sense of immersion and realism by combining virtual visual and audio cues with real sights and sounds. It may therefore be desirable to present digital sounds to a user of an XR system so that the sounds seem to occur naturally and are consistent with the user's expectations of sounds in the user's real environment. Generally, users expect that virtual sounds will take on the acoustic properties of the real environment in which they are heard. For example, a user of an XR system in a large concert hall would expect the virtual sounds of the XR system to have a cavernous sonic quality, and conversely, a user in a small apartment would expect the sounds to be more attenuated, close, and immediate. In addition to matching the acoustic properties of the virtual sounds with the real and / or virtual environment, realism is further enhanced by spatializing the virtual sounds. For example, a virtual object may visually fly over the user from behind, and the user may expect corresponding virtual sounds to similarly reflect the spatial movement of the virtual object relative to the user.

[0005] Existing technologies often miss these expectations, such as by presenting virtual audio that does not take into account the user's surroundings and does not accommodate the spatial movement of virtual objects, leading to a sense of inauthenticity that can detract from the user experience. Observations of users of XR systems indicate that while users can be relatively tolerant of visual inconsistencies between virtual content and the real environment (e.g., inconsistencies in lighting), they can be more sensitive to auditory inconsistencies. Our unique auditory experiences, continually refined throughout our lives, can make us acutely aware of how our physical environment affects the sounds we hear, and we can be highly sensitive to sounds that do not match those expectations. In XR systems, such inconsistencies can be uncomfortable, turning an immersive and engaging experience into a gimmicky imitation. In extreme examples, auditory inconsistencies can cause motion sickness and other adverse effects due to the inability of the inner ear to reconcile an auditory stimulus with its corresponding visual cue.

[0006] A system architecture is needed to organize and manage the system for generating virtual audio. Generating virtual audio may involve managing and storing information about the user's environment so that the information can be used to produce realistic virtual sounds. The audio system architecture may therefore need to interface with other systems to receive and utilize information related to the audio engine. Furthermore, it may be desirable to have an audio system architecture that can present realistic sounds without interruption during use. An audio system architecture that can update the audio engine without interruption in use can produce an immersive user experience in which auditory signals continually reflect the user's environment.

[0007] By taking into account the characteristics of the user's physical environment, the systems and methods described herein can simulate what the user would hear as if the virtual sounds were real sounds naturally generated in that environment. By presenting virtual sounds in a manner faithful to the way sounds behave in the real world, the user may experience an increased sense of connection with the mixed reality environment. Similarly, by presenting location-aware virtual content that responds to the user's movements and environment, the content becomes more subjective, interactive, and realistic; for example, a user's experience at point A may be entirely different from his or her experience at point B. This enhanced realism and interaction can provide the foundation for new applications of mixed reality, such as those that use spatially aware audio to enable novel forms of gameplay, social features, or interactive behaviors. Summary of the Invention [Means for solving the problem]

[0008] Disclosed herein are systems and methods for storing, organizing, and maintaining acoustic data for a mixed reality system. The system may include one or more sensors of a head-mounted device, a speaker of the head-mounted device, and one or more processors configured to execute the method. The method for execution by the one or more processors may include receiving a request to present an audio signal. An environment may be identified via one or more sensors of the head-mounted device. One or more audio model components associated with the environment may be retrieved. A first audio model may be generated based on the audio model components. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and based on the request to present the audio signal. The modified audio signal may be presented via the speaker of the head-mounted device.

[0009] In some system embodiments, the second audio model may be generated by an audio service. In some system embodiments, the modified audio signal may be determined by an audio service. In some system embodiments, the second audio model may be a replica of the first audio model.

[0010] In some system embodiments, the one or more audio model components may include one or more dimensions of the environment. In some system embodiments, the one or more audio model components may include reverberation time. In some system embodiments, the one or more audio model components may include reverberation gain. In some system embodiments, the one or more audio model components may include a transmission loss coefficient. In some system embodiments, the one or more audio model components may include an absorption coefficient.

[0011] The method may include receiving a request to present an audio signal. An environment may be identified via one or more sensors of the head-worn device. One or more audio model components associated with the environment may be retrieved. A first audio model may be generated based on the audio model components. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and based on the request to present the audio signal. The modified audio signal may be presented via a speaker of the head-worn device.

[0012] In some method embodiments, the second audio model may be generated by an audio service. In some method embodiments, the modified audio signal may be determined by an audio service. In some method embodiments, the second audio model may be a replica of the first audio model.

[0013] In some method embodiments, the one or more audio model components may include one or more dimensions of the environment. In some method embodiments, the one or more audio model components may include a reverberation time. In some method embodiments, the one or more audio model components may include a reverberation gain. In some method embodiments, the one or more audio model components may include a transmission loss coefficient. In some method embodiments, the one or more audio model components may include an absorption coefficient.

[0014] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method for execution by the one or more processors may include receiving a request to present an audio signal, identifying an environment via one or more sensors of a head-worn device, retrieving one or more audio model components associated with the environment, generating a first audio model based on the audio model components, generating a second audio model based on the first audio model, determining a modified audio signal based on the second audio model and based on the request to present the audio signal, and presenting the modified audio signal via a speaker of the head-worn device.

[0015] In some non-transient computer readable media embodiments, the second audio model may be generated by an audio service. In some non-transient computer readable media embodiments, the modified audio signal may be determined by an audio service. In some non-transient computer readable media embodiments, the second audio model may be a replica of the first audio model.

[0016] In some non-transient computer readable media embodiments, the one or more audio model components may include one or more dimensions of the environment. In some non-transient computer readable media embodiments, the one or more audio model components may include a reverberation time. In some non-transient computer readable media embodiments, the one or more audio model components may include a reverberation gain. In some non-transient computer readable media embodiments, the one or more audio model components may include a transmission loss coefficient. In some non-transient computer readable media embodiments, the one or more audio model components may include an absorption coefficient. The present invention provides, for example, the following: (Item 1) 1. A system comprising: one or more sensors in a head-worn device; a speaker of the head-worn device; and one or more processors, said one or more processors comprising: Receiving a request to present an audio signal; Identifying an environment via one or more sensors of the head-worn device; Retrieving one or more audio model components associated with the environment; and generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and based on a request for presenting an audio signal; presenting the modified audio signal through a speaker of the head-worn device; and one or more processors configured to execute a method comprising: A system comprising: (Item 2) 2. The system of claim 1, wherein the second audio model is generated by an audio service. (Item 3) 2. The system of claim 1, wherein the modified audio signal is determined by an audio service. (Item 4) 2. The system of claim 1, wherein the second audio model is a replica of the first audio model. (Item 5) 2. The system of claim 1, wherein the one or more audio model components comprise one or more dimensions of the environment. (Item 6) 2. The system of claim 1, wherein the one or more audio model components comprise a reverberation time. (Item 7) 2. The system of claim 1, wherein the one or more audio model components comprise a reverberation gain. (Item 8) 2. The system of claim 1, wherein the one or more audio model components comprise a transmission loss coefficient. (Item 9) 2. The system of claim 1, wherein the one or more audio model components comprise absorption coefficients. (Item 10) 1. A method comprising: Receiving a request to present an audio signal; Identifying an environment via one or more sensors of a head-worn device; Retrieving one or more audio model components associated with the environment; and generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and based on a request for presenting an audio signal; presenting the modified audio signal through a speaker of the head-worn device; and A method comprising: (Item 11) Item 11. The method of item 10, wherein the second audio model is generated by an audio service. (Item 12) Item 11. The method of item 10, wherein the modified audio signal is determined by an audio service. (Item 13) Item 11. The method of item 10, wherein the second audio model is a replica of the first audio model. (Item 14) 11. The method of claim 10, wherein the one or more audio model components comprise one or more dimensions of the environment. (Item 15) Item 11. The method of item 10, wherein the one or more audio model components comprise a reverberation time. (Item 16) Item 11. The method of item 10, wherein the one or more audio model components comprise a reverberation gain. (Item 17) Item 11. The method of item 10, wherein the one or more audio model components comprise transmission loss coefficients. (Item 18) Item 11. The method of item 10, wherein the one or more audio model components comprise absorption coefficients. (Item 19) A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: Receiving a request to present an audio signal; Identifying an environment via one or more sensors of a head-worn device; Retrieving one or more audio model components associated with the environment; and generating a first audio model based on the audio model components; generating a second audio model based on the first audio model; determining a modified audio signal based on the second audio model and based on a request for presenting an audio signal; presenting the modified audio signal through a speaker of the head-worn device; and A non-transitory computer readable medium for carrying out a method comprising: (Item 20) 20. The non-transitory computer-readable medium of claim 19, wherein the one or more audio model components comprise one or more dimensions of the environment. [Brief description of the drawings]

[0017] [Figure 1A] 1A-1C illustrate an example mixed reality environment, according to some embodiments. [Figure 1B] 1A-1C illustrate an example mixed reality environment, according to some embodiments. [Figure 1C] 1A-1C illustrate an example mixed reality environment, according to some embodiments.

[0018] [Figure 2A] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 2B] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 2C] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 2D] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments.

[0019] [Figure 3A] FIG. 3A illustrates an example mixed reality handheld controller that can be used to provide input to a mixed reality environment, according to some embodiments.

[0020] [Figure 3B] FIG. 3B illustrates an example auxiliary unit that may be used with an example mixed reality system, according to some embodiments.

[0021] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example mixed reality system, according to some embodiments.

[0022] [Diagram 5] FIG. 5 illustrates an example of a virtual audio system, according to some embodiments.

[0023] [Figure 6] FIG. 6 illustrates an example process for updating an audio model according to some embodiments.

[0024] [Figure 7] FIG. 7 illustrates an example process for updating an audio model according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Detailed Description In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0026] Mixed Reality Environment

[0027] Like all people, users of a mixed reality system exist in a real environment, i.e., the three-dimensional portion of the "real world" and all of its content are perceivable by the user. For example, the user perceives the real environment using normal human senses, i.e., sight, hearing, touch, taste, and smell, and interacts with the real environment by moving his or her body within the real environment. Locations within the real environment can be described as coordinates in a coordinate space. For example, coordinates can include latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, a vector can describe a quantity, having a direction and magnitude in a coordinate space.

[0028] A computing device may maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computed representation of a three-dimensional space. A virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, a circuit (e.g., a processor) of a computing device may maintain and update a state of the virtual environment. That is, the processor may determine a state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, coefficient of friction), and input received from a user indicates that a force should be applied to the object in a certain directional vector, the processor may apply the laws of kinematics and use basic mechanics to determine the location of the object at time t1. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to creating and deleting virtual objects in the virtual environment, software (e.g., scripts) for defining behavior of virtual objects or characters in the virtual environment, software for defining behavior of signals (e.g., audio signals) in the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals in the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving a virtual object over time), or many other possibilities.

[0029] An output device, such as a display or speaker, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that may be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera," with its origin coordinates, viewing axis, and frustum) and render on the display a viewable scene of the virtual environment corresponding to that view. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that may be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment may generate sounds originating from the object's location coordinates (e.g., a virtual character may speak or create a sound effect), or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. The processor can determine audio signals corresponding to "listener" coordinates, e.g., audio signals corresponding to the synthesis of sounds in the virtual environment and that are mixed and processed to simulate the audio signals that would be heard by a listener at the listener coordinates, and present the audio signals to the user via one or more speakers.

[0030] Because the virtual environment exists only as a computational structure, the user cannot directly perceive the virtual environment using normal senses. Instead, the user can only indirectly perceive the virtual environment, as presented to the user, for example, by a display, a speaker, a tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data via input devices or sensors to a processor, which can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond appropriately within the virtual environment.

[0031] A mixed reality system can present a user with a mixed reality environment ("MRE") that combines aspects of real and virtual environments, for example, using a see-through display and / or one or more speakers (which may be incorporated, for example, into a wearable head device). In some embodiments, the one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space. In some examples, the real coordinate space and the corresponding virtual coordinate space are related to each other by a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with the transformation matrix) may define a first location in the real environment and also a second corresponding location in the virtual environment, and vice versa.

[0032] In an MRE, a virtual object (e.g., in a virtual environment associated with the MRE) may correspond to a real object (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamppost (a real object) at a location coordinate, the virtual environment of the MRE may include a virtual lamppost (a virtual object) at a corresponding location coordinate. As used herein, a real object combines with its corresponding virtual object to comprise a "mixed reality object." It is not necessary for a virtual object to perfectly match or match a corresponding real object. In some examples, a virtual object may be a simplified version of a corresponding real object. For example, if the real environment includes a real lamppost, the corresponding virtual object may include a cylinder of approximately the same height and radius as the real lamppost (reflecting that a lamppost may be approximately cylindrical in shape). Simplifying virtual objects in this manner may enable computational efficiencies and simplify calculations to be performed on such virtual objects. Furthermore, in some examples of an MRE, not all real objects in the real environment may be associated with a corresponding virtual object. Similarly, in some embodiments of the MRE, not all virtual objects in the virtual environment may be associated with corresponding real objects, i.e., some virtual objects may exist solely within the virtual environment of the MRE without any real-world counterpart.

[0033] In some embodiments, virtual objects may have characteristics that differ, sometimes significantly, from those of the corresponding real object. For example, a real environment in an MRE may contain a green, two-pronged cactus, a thorny, inanimate object, while the corresponding virtual object in the MRE may have the characteristics of a green, two-armed virtual character with human facial features and a surly attitude. In this embodiment, the virtual object resembles its corresponding real object in some characteristics (color, number of arms) but differs from the real object in other characteristics (facial features, personality). In this way, virtual objects have the potential to represent real objects in creative, abstract, exaggerated, or fictitious ways, or to impart behaviors (e.g., human personality) to otherwise inanimate real objects. In some embodiments, virtual objects may be purely fictitious creations with no real-world counterpart (e.g., virtual monsters in a virtual environment, perhaps in a location that corresponds to a void in the real environment).

[0034] Compared to a VR system that presents a virtual environment to a user while obscuring the real environment, a mixed reality system that presents an MRE offers the advantage that the real environment remains perceptible while the virtual environment is presented. Thus, a user of a mixed reality system can experience and interact with a corresponding virtual environment using visual and audio cues associated with the real environment. As an example, a user of a VR system may struggle to perceive or interact with a virtual object displayed in the virtual environment because, as mentioned above, the user cannot directly perceive or interact with the virtual environment, whereas a user of an MR system may find it intuitive and natural to interact with a virtual object by seeing, hearing, and touching the corresponding real object in his or her real environment. This level of interaction may enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by presenting a real environment and a virtual environment simultaneously, a mixed reality system may reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with a VR system. Mixed reality systems also offer many possibilities for applications that can augment or modify our experience of the real world.

[0035] 1A illustrates an exemplary real environment 100 in which a user 110 uses a mixed reality system 112. The mixed reality system 112 may include a display (e.g., a see-through display) and one or more speakers, and one or more sensors (e.g., cameras), for example, as described below. The illustrated real environment 100 includes a rectangular room 104A in which the user 110 is standing, and real objects 122A (lamp), 124A (table), 126A (sofa), and 128A (painting). The room 104A further includes a location coordinate 106, which may be considered as the origin of the real environment 100. As shown in FIG. 1A, an environment / world coordinate system 108 (with an x-axis 108X, a y-axis 108Y, and a z-axis 108Z), with its origin at point 106 (world coordinates), may define a coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 may correspond to where the mixed reality system 112 was powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 may be reset during operation. In some examples, the user 110 may be considered a real object in the real environment 100. Similarly, the body parts (e.g., hands, feet) of the user 110 may be considered real objects in the real environment 100. In some examples, the user / listener / head coordinate system 114 (comprising an x-axis 114X, a y-axis 114Y, and a z-axis 114Z), with its origin at point 115 (e.g., user / listener / head coordinate), may define a coordinate space for the user / listener / head on which the mixed reality system 112 is located. The origin 115 of the user / listener / head coordinate system 114 may be defined relative to one or more components of the mixed reality system 112. For example, the origin 115 of the user / listener / head coordinate system 114 may be defined relative to the display of the mixed reality system 112, such as during an initial calibration of the mixed reality system 112. Matrices (which may include translation matrices and quaternion or other rotation matrices) or other suitable representations can characterize the transformation between the user / listener / head coordinate system 114 space and the environment / world coordinate system 108 space.In some embodiments, the left ear coordinates 116 and right ear coordinates 117 may be defined relative to the origin 115 of the user / listener / head coordinate system 114. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / listener / head coordinate system 114 space. The user / listener / head coordinate system 114 can simplify the representation of location relative to the user's head or head-mounted device, e.g., relative to the environment / world coordinate system 108. Using simultaneous localization and mapping (SLAM), visual odometry, or other techniques, the transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time.

[0036] 1B illustrates an exemplary virtual environment 130 that corresponds to the real environment 100. The virtual environment 130 shown includes a virtual rectangular room 104B that corresponds to the real rectangular room 104A, a virtual object 122B that corresponds to the real object 122A, a virtual object 124B that corresponds to the real object 124A, and a virtual object 126B that corresponds to the real object 126A. Metadata associated with the virtual objects 122B, 124B, 126B may include information derived from the corresponding real objects 122A, 124A, 126A. The virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in the real environment 100. A real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate system 133 (with x-axis 133X, y-axis 133Y, and z-axis 133Z) with its origin at point 134 (persistent coordinate) may define a coordinate space for the virtual content. The origin 134 of the persistent coordinate system 133 may be defined relative to / with respect to one or more real objects, such as real object 126A. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the persistent coordinate system 133 space and the environment / world coordinate system 108 space. In some embodiments, virtual objects 122B, 124B, 126B, and 132 may each have its own persistent coordinate point relative to the origin 134 of the persistent coordinate system 133. In some embodiments, there may be multiple persistent coordinate systems, and virtual objects 122B, 124B, 126B, and 132 may each have its own persistent coordinate point relative to one or more persistent coordinate systems.

[0037] 1A and 1B, the environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the illustrated embodiment, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Thus, a first location in the real environment 100 and a second corresponding location in the virtual environment 130 can be described with respect to the same coordinate space. This simplifies the steps of identifying and displaying corresponding locations in the real and virtual environments, since the same coordinates can be used to identify both locations. However, in some embodiments, the corresponding real and virtual environments need not use a shared coordinate space. For example, in some embodiments (not shown), matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0038] 1C illustrates an exemplary MRE 150 that simultaneously presents aspects of real environment 100 and virtual environment 130 to user 110 via mixed reality system 112. In the example shown, MRE 150 simultaneously presents to user 110 real objects 122A, 124A, 126A, and 128A from real environment 100 (e.g., via a translucent portion of a display of mixed reality system 112) and virtual objects 122B, 124B, 126B, and 132 from virtual environment 130 (e.g., via a display active display portion of mixed reality system 112). As described above, origin 106 acts as the origin for a coordinate space corresponding to MRE 150, and coordinate system 108 defines the x-, y-, and z-axes for the coordinate space.

[0039] In the illustrated example, the mixed reality objects include corresponding pairs of real and virtual objects (i.e., 122A / 122B, 124A / 124B, 126A / 126B) that occupy corresponding locations in coordinate space 108. In some examples, both real and virtual objects may be visible to user 110 at the same time. This may be desirable in instances where, for example, a virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where a virtual object presents a missing portion of an ancient damaged statue). In some examples, the virtual objects (122B, 124B, and / or 126B) may be displayed so as to occlude the corresponding real objects (122A, 124A, and / or 126A) (e.g., via active pixelated occlusion using a pixelated occlusion shutter). This may be desirable, for example, in instances where a virtual object acts as a visual replacement for a corresponding real object (such as in interactive storytelling applications where inanimate real objects become "living" characters).

[0040] In some examples, real objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data that does not necessarily constitute a virtual object. The virtual content or helper data may facilitate processing or handling of the virtual object within a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information may enable or facilitate calculations involving the real object without incurring unnecessary computational overhead.

[0041] In some embodiments, the presentation described above may also incorporate audio aspects. For example, in MRE 150, virtual monster 132 may be associated with one or more audio signals, such as footstep effects, that are generated as the monster walks around MRE 150. As described further below, a processor in mixed reality system 112 may calculate an audio signal corresponding to a mixed and processed combination of all such sounds within MRE 150 and present the audio signal to user 110 via one or more speakers included within mixed reality system 112 and / or one or more external speakers.

[0042] Exemplary Mixed Reality System

[0043] An exemplary mixed reality system 112 may include a wearable head device (e.g., a wearable augmented reality or mixed reality head device) with a display (which may include left and right see-through displays, which may be eyepiece displays, and associated components for coupling light from the displays to the user's eyes), left and right speakers (e.g., positioned adjacent the user's left and right ears, respectively), an inertial measurement unit (IMU) (e.g., mounted on a temple arm of the head device), a quadrature coil electromagnetic receiver (e.g., mounted on the left temple part), left and right cameras (e.g., depth (time of flight) cameras) oriented away from the user, and left and right eye cameras (e.g., for detecting the user's eye movements) oriented toward the user. However, the mixed reality system 112 may incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic). In addition, the mixed reality system 112 may incorporate networking features (e.g., Wi-Fi capabilities) to communicate with other devices and systems, including other mixed reality systems. The mixed reality system 112 may further include a battery (which may be mounted in an auxiliary unit, such as a belt pack designed to be worn around the waist of a user), a processor, and a memory. The wearable head device of the mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to the processor to implement simultaneous localization and mapping (SLAM) and / or visual odometry algorithms. In some examples, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt pack, as described further below.

[0044] 2A-2D illustrate components of an example mixed reality system 200 (which may correspond to the mixed reality system 112) that may be used to present an MRE (which may correspond to the MRE 150) or other virtual environment to a user. FIG. 2A illustrates a perspective view of a wearable head device 2102 included in the example mixed reality system 200. FIG. 2B illustrates a top view of the wearable head device 2102 mounted on a user's head 2202. FIG. 2C illustrates a front view of the wearable head device 2102. FIG. 2D illustrates an edge view of an example eyepiece 2110 of the wearable head device 2102. As shown in FIGS. 2A-2C, the example wearable head device 2102 includes an example left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an example right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 can include a transmissive element through which the real environment is visible, and a display element for presenting a display (e.g., via image-wise modulated light) that is superimposed on the real environment. In some embodiments, such a display element can include a surface diffractive optical element for controlling the flow of image-wise modulated light. For example, the left eyepiece 2108 can include a left internal coupling grating set 2112, a left orthogonal pupil expansion (OPE) grating set 2120, and a left exit (output) pupil expansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 can include a right internal coupling grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. The image-wise modulated light can be transferred to the user's eye via the internal coupling gratings 2112 and 2118, the OPE 2114 and 2120, and the EPE 2116 and 2122. Each internal coupling grating set 2112, 2118 can be configured to deflect light towards its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to progressively deflect light downward towards its associated EPE 2122, 2116, thereby extending the exit pupil formed horizontally.Each EPE 2122, 2116 can be configured to progressively redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outwardly to a user eyebox location (not shown), defined behind the eyepieces 2108, 2110, to vertically extend an exit pupil formed in the eyebox. Alternatively, instead of the internal coupling grating sets 2112 and 2118, the OPE grating sets 2114 and 2120, and the EPE grating sets 2116 and 2122, the eyepieces 2108 and 2110 can include gratings and / or other arrangements of refractive and reflective features to control the coupling of the image-wise modulated light into the user's eye.

[0045] In some embodiments, the wearable head device 2102 can include a left temple arm 2130 and a right temple arm 2132, with the left temple arm 2130 including a left speaker 2134 and the right temple arm 2132 including a right speaker 2136. A quadrature coil electromagnetic receiver 2138 can be located in the left temple piece or another suitable location in the wearable head unit 2102. An inertial measurement unit (IMU) 2140 can be located in the right temple arm 2132 or another suitable location in the wearable head device 2102. The wearable head device 2102 can also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142, 2144 can be preferably oriented in different directions so that both cover a wider field of view.

[0046] 2A-2D, a left source of image-wise modulated light 2124 can be optically coupled into the left eyepiece 2108 through a left internal coupling grating set 2112, and a right source of image-wise modulated light 2126 can be optically coupled into the right eyepiece 2110 through a right internal coupling grating set 2118. The source of image-wise modulated light 2124, 2126 can include, for example, a fiber optic scanner, a projector including an electronic light modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator, or an emissive display such as a micro light emitting diode (μLED) or micro organic light emitting diode (μOLED) panel that is coupled into the internal coupling grating sets 2112, 2118 using one or more lenses per side. The input coupling grating sets 2112, 2118 can deflect light from the source of image-wise modulated light 2124, 2126 to an angle above the critical angle for total internal reflection (TIR) ​​for the eyepieces 2108, 2110. The OPE grating sets 2114, 2120 progressively deflect the propagating light downwards by TIR towards the EPE grating sets 2116, 2122. The EPE grating sets 2116, 2122 progressively couple the light towards the user's face, including the pupils of the user's eyes.

[0047] In some examples, as shown in FIG. 2D , the left eyepiece 2108 and the right eyepiece 2110 each include multiple waveguides 2402. For example, each eyepiece 2108, 2110 can include multiple individual waveguides, each dedicated to a separate color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 can include multiple sets of such waveguides, each set configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex with respect to the user's eye, for example, to present a virtual object located at a distance in front of the user (e.g., a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating sets 2116, 2122 can include curved grating grooves to provide a convex wavefront curvature by modifying the Poynting vector of the light exiting across each EPE.

[0048] In some examples, stereoscopically accommodated left and right eye images can be presented to the user through light modulators 2124, 2126 and eyepieces 2108, 2110 for each image to create the perception that the displayed content is three-dimensional. The perceived realism of the presentation of three-dimensional virtual objects can be enhanced by selecting the waveguides (and thus the corresponding wavefront curvatures) such that the virtual objects are displayed at distances that approximate the distances shown by the stereoscopic left and right images. The technique can also reduce motion sickness suffered by some users, which can be caused by differences between the depth perception cues provided by the stereoscopic left and right eye images and the automatic accommodation (e.g., object distance-dependent focus) of the human eye.

[0049] FIG. 2D illustrates an edge view from the top of the right eyepiece 2110 of the exemplary wearable head device 2102. As shown in FIG. 2D, the plurality of waveguides 2402 can include a first subset of three waveguides 2404 and a second subset of three waveguides 2406. The two subsets of waveguides 2404, 2406 can be distinguished by different EPE gratings that feature different grating line curvatures to impart different wavefront curvatures to the exiting light. Within each of the subsets of waveguides 2404, 2406, each waveguide can be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. (Although not shown in FIG. 2D, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.)

[0050] 3A illustrates an example handheld controller component 300 of a mixed reality system 200. In some examples, the handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, the buttons 350 may be configured for use as an optical tracking target to track six degrees of freedom (6DOF) movement of the handheld controller 300, for example, in conjunction with a camera or other optical sensor (which may be mounted in a head unit (e.g., wearable head device 2102) of the mixed reality system 200). In some examples, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting a position or orientation, such as a position or orientation relative to the wearable head device 2102. In some examples, such a tracking component may be positioned in a handle of the handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 can be configured to provide one or more output signals corresponding to one or more of a button press state, or a position, orientation, and / or movement (e.g., via an IMU) of the handheld controller 300. Such output signals may be used as inputs to a processor of the mixed reality system 200. Such inputs may correspond to the position, orientation, and / or movement of the handheld controller (or, for that matter, the position, orientation, and / or movement of a user's hand holding the controller). Such inputs may also correspond to a user pressing a button 350.

[0051] 3B illustrates an example auxiliary unit 320 of the mixed reality system 200. The auxiliary unit 320 can include a battery for providing energy and operating the system 200, and can include a processor for executing programs and operating the system 200. As shown, the example auxiliary unit 320 includes a clip 2128 for attaching the auxiliary unit 320 to a user's belt, etc. It will be apparent that other form factors are also suitable for the auxiliary unit 320, including form factors that do not involve mounting the unit on a user's belt. In some embodiments, the auxiliary unit 320 is coupled to the wearable head device 2102 through a multi-tube cable, which may include, for example, electrical wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 can also be used.

[0052] In some examples, the mixed reality system 200 can include one or more microphones to detect sounds and provide corresponding signals to the mixed reality system. In some examples, the microphones may be attached to or integrated with the wearable head device 2102 and configured to detect the user's voice. In some examples, the microphones may be attached to or integrated with the handheld controller 300 and / or the auxiliary unit 320. Such microphones may be configured to detect environmental sounds, ambient noise, the user's or a third party's voice, or other sounds.

[0053] FIG. 4 illustrates an example functional block diagram that may correspond to an example mixed reality system, such as the mixed reality system 200 described above (which may correspond to the mixed reality system 112 with respect to FIG. 1). As shown in FIG. 4, the example handheld controller 400B (which may correspond to the handheld controller 300 ("totem")) includes a totem / wearable head device six degree of freedom (6DOF) totem subsystem 404A, and the example wearable head device 400A (which may correspond to the wearable head device 2102) includes a totem / wearable head device 6DOF subsystem 404B. In an example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translational directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three translational offsets may be represented as X, Y, and Z offsets in such a coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations, as a rotation matrix, as a quaternion, or some other representation. In some examples, the wearable head device 400A, one or more depth cameras 444 (and / or one or more non-depth cameras) included within the wearable head device 400A, and / or one or more optical targets (e.g., buttons 350 of handheld controller 400B as described above or dedicated optical targets included within handheld controller 400B) may be used for 6DOF tracking. In some examples, the handheld controller 400B may include a camera as described above, and the wearable head device 400A may include an optical target for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly transmit and receive three distinguishable signals.By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined. Additionally, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.

[0054] In some examples, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to an inertial coordinate space (e.g., a coordinate space fixed relative to the real environment), e.g., to compensate for movement of the wearable head device 400A relative to the coordinate system 108. For example, such a transformation may be necessary so that the display of the wearable head device 400A presents virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device), rather than in a fixed position and orientation on the display (e.g., the same position in the bottom right corner of the display), preserving the illusion that the virtual objects are present in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensation transformation between coordinate spaces can be determined by processing images from the depth camera 444 using SLAM and / or visual odometry procedures to determine a transformation of the wearable head device 400A relative to the coordinate system 108. In the example shown in FIG. 4, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine a position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and another coordinate space (e.g., an inertial coordinate space). Similarly, in some examples, an additional source of information regarding the user's head pose and location is obtained from the IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding fast adjustments of the user's head pose and position.

[0055] In some examples, the depth camera 444 can provide 3D images to a hand gesture tracker 411, which can be implemented within a processor of the wearable head device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from the depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0056] In some embodiments, one or more processors 416 may be configured to receive data from the 6DOF headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, and / or the hand gesture tracker 411 of the wearable head device. The processor 416 may also send and receive control signals to and from the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components, such as an audio / visual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the imagewise modulated light sources 424, 426, for example, as described above with respect to FIGS. 2A-2D. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (e.g., which may be moved by the user via the handheld controller 320). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0057] 4, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included in an auxiliary unit 400C (which may correspond to auxiliary unit 320 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or provide power to the wearable head device 400A or handheld controller 400B. Including such components in an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0058] 4 presents elements corresponding to various components of an exemplary mixed reality system, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements presented in FIG. 4 as being associated with the auxiliary unit 400C may instead be associated with the wearable head device 400A or the handheld controller 400B. Furthermore, some mixed reality systems may dispense with the handheld controller 400B or the auxiliary unit 400C entirely. Such variations and modifications should be understood as falling within the scope of the disclosed embodiments.

[0059] Environmental Sound Persistence

[0060] As described above, an MRE (as experienced via a mixed reality system, e.g., mixed reality system 112, which may include components such as wearable head unit 200, handheld controller 300, or auxiliary unit 320, described above) can present to a user of the MRE audio signals that appear to originate at a sound source with origin coordinates within the MRE. That is, the user may perceive these audio signals as if they were real audio signals originating from the origin coordinates of the sound source.

[0061] In some cases, audio signals may be considered virtual in that they correspond to calculated signals in a virtual environment. Virtual audio signals can be presented to the user as real audio signals detectable by the human ear, for example, as produced via speakers 2134 and 2136 of the wearable head unit 200 in FIG.

[0062] The sound source may correspond to a real object and / or a virtual object. For example, a virtual object (e.g., virtual monster 132 of FIG. 1C) may emit an audio signal in the MRE, which is represented in the MRE as a virtual audio signal and presented to the user as a real audio signal. For example, virtual monster 132 of FIG. 1C may emit a virtual sound corresponding to the monster's speech (e.g., dialogue) or a sound effect. Similarly, a real object (e.g., real object 122A of FIG. 1C) may be made to appear to emit a virtual audio signal in the MRE, which is represented in the MRE as a virtual audio signal and presented to the user as a real audio signal. For example, real lamp 122A may emit a virtual sound corresponding to the sound effect of a lamp being switched on or off, even if the lamp is not switched on or off in the real environment. The virtual sound may correspond to the position and orientation of the sound source (whether real or virtual). For example, if a virtual sound is presented to a user as a real audio signal (e.g., via speakers 2134 and 2136), the user may perceive the virtual sound to originate from the location of the sound source. A sound source is referred to herein as a "virtual sound source" even though the underlying object that is apparently made to emit sound may itself correspond to a real or virtual object as described above.

[0063] Some virtual or mixed reality environments suffer from the perception that the environment does not feel real or authentic. One reason for this perception is that audio and visual cues do not always match each other in such environments. For example, if a user is positioned behind a large brick wall in an MRE, the user may expect sounds coming from behind the brick wall to be quieter and muffled than sounds coming from immediately next to the user. This expectation is based on the user's auditory experience in the real world, where sounds become quieter and muffled when passing behind a large dense object. When the user is presented with an audio signal that is said to come from behind the brick wall, but is not muffled and is presented at full volume, the illusion that the sound comes from behind the brick wall is ruined. The entire virtual experience may feel fake and inauthentic, in part because it does not match the user's expectations based on real-world interactions. Furthermore, in some cases, the "uncanny valley" problem arises, in which even slight differences between the virtual and real experiences can cause an increased sense of discomfort. It is desirable to improve the user's experience by presenting audio signals within the MRE that appear to interact, even slightly, realistically with objects in the user's environment. The more consistent such audio signals are with the user's expectations, based on real-world experiences, the more immersive and engaging the user's experience within the MRE can be.

[0064] One way that a user perceives and understands the environment around them is through audio cues. In the real world, real audio signals heard by a user are affected by the location where those audio signals originate and the objects with which they interact. For example, all other factors being equal, a sound originating at a distance away from the user (e.g., a dog barking in the distance) will appear quieter than the same sound originating at a short distance from the user (e.g., a dog barking in the same room as the user). The user can therefore identify the location of the dog in the real environment based in part on the perceived volume of the barking sound. Similarly, other factors being equal, a sound traveling away from the user (e.g., the voice of a person facing away from the user) will appear less clear and more muffled (i.e., low-pass filtered) than the same sound traveling towards the user (e.g., the voice of a person facing towards the user). The user can therefore identify the orientation of a person in the real environment based on the perceived characteristics of that person's voice.

[0065] A user's perception of a real audio signal may also be affected by the presence of objects in the environment with which the audio signal interacts. That is, a user may perceive not only the audio signal produced by a sound source, but also the reflection of that audio signal on nearby objects and the reverberant signature imparted by the surrounding acoustic space. For example, if a person is speaking in a small room with nearby walls, those walls may produce short natural reverberant signals that occur as the person's voice reflects off the walls. A user may infer from those reverberations that they are in a small room with nearby walls. Similarly, a large concert hall or cathedral may produce longer reverberations from which a user may infer that they are in a large, spacious room. Similarly, reverberations of audio signals may take on different sonic characteristics based on the position or orientation of the surfaces against which they reflect, or the materials of those surfaces. For example, reverberations against tiled walls will sound different than reverberations against brick, carpet, drywall, or other materials. These reverberation characteristics can be used by users to acoustically understand the size, shape, and material composition of the space in which they reside.

[0066] The above examples illustrate how audio cues can inform a user's perception of their surrounding environment. These cues can act in combination with visual cues, for example, if a user sees a distant dog, the user may expect the dog's barking to match that distance (if this is not the case, the user may become confused or disoriented, as in some virtual environments). In some examples, visual cues may be limited or unavailable, such as in low light environments or for vision-impaired users, in which case audio cues may take on particular importance and serve as the user's primary means of understanding their environment.

[0067] A system architecture may be useful for organizing, storing, recalling, and / or managing information required to present realistic virtual audio. For example, an MR system (e.g., MR system 112, 200) may manage environmental information such as the real environment in which a user may be present, the acoustic properties that the real environment may have, and / or the location within the real environment where the user may be located. The MR system may further manage information about objects in the real and / or virtual environment (e.g., objects that may affect the general acoustic properties of the real environment and / or objects that may affect the acoustic properties of virtual sound sources that interact with the object). The MR system may also manage information about virtual sound sources. For example, the location where a virtual sound source is located may be relevant in rendering realistic virtual audio.

[0068] In addition to managing the virtual audio system, it may also be necessary to simultaneously manage other systems to present a full MR experience. For example, a full MR experience may require a virtual vision system, which may manage information used to render virtual objects. A full MR experience may require a simultaneous localization and mapping system ("SLAM"), which may build, update, and / or maintain a three-dimensional model of the user's environment. The MR system (e.g., MR systems 112, 200) may manage these systems and present the full MR experience in addition to the virtual audio system. A virtual audio system architecture may be useful for managing the interactions between these systems and facilitating data transfer, management, storage, and / or security.

[0069] In some embodiments, the system (e.g., a virtual audio system) may interact with other higher level systems. In some embodiments, a lower level system (e.g., a virtual audio system) may interact more closely with hardware level inputs and / or outputs, while a higher level system (e.g., an application) may interface with a lower level system. A higher level system may utilize a lower level system to perform its function (e.g., a gaming application may rely on a lower level virtual audio system to render realistic virtual audio). A virtual audio system may benefit from a system architecture designed to manage interactions with higher level systems while maintaining the integrity of the virtual audio system. For example, multiple higher level systems (e.g., multiple third party applications) may interface with a virtual audio system simultaneously or substantially simultaneously. In some embodiments, maintaining a single virtual audio system that can render virtual audio may be more computationally efficient than having each higher level system maintain a separate audio system. For example, in some embodiments, a single digital reverberator may be used to process sound objects from multiple higher level systems when those objects are intended to exist within the same virtual or real acoustic space (e.g., a room in which a user resides). A well-designed system architecture may also protect the integrity of information (e.g., from data corruption and / or tampering) that may be used in other applications.

[0070] In some embodiments, it may be advantageous to design the system architecture so that changes can be made in real time without disrupting service to other systems (e.g., higher-level systems). For example, a virtual audio system may store, maintain, or otherwise manage an audio model that takes into account the acoustic properties of a real environment (e.g., a room). When a user changes the real environment (e.g., moves to a different room), the audio model may be updated to account for the changes in the real environment. If the MR system is currently in use (e.g., the MR system is presenting virtual visuals and / or virtual audio to the user), it may be necessary to update the audio model when a different system (e.g., a higher-level system) is still using the audio model to render virtual audio.

[0071] In some embodiments, it may be advantageous to design the system architecture to propagate changes to other systems. For example, some systems may maintain separate copies of the audio model, or some systems may store certain repeated sound effects that are rendered using the audio model maintained by the virtual audio system. Thus, it may be advantageous to propagate changes made to the virtual audio system to other systems. For example, if a user changes the environment (e.g., moves rooms) and a new audio model may be more accurate, the virtual audio system may modify its audio model and notify any clients (e.g., systems that may use and / or rely on the virtual audio system) of the change. In some embodiments, the clients may then query the virtual audio system and update their internal data accordingly.

[0072] FIG. 5 illustrates an exemplary virtual audio system, according to some embodiments. The virtual audio system 500 can include a persistent module 502. A module (e.g., persistent module 502) can include one or more computer systems configured to execute instructions and / or store one or more data structures. In some embodiments, a module (e.g., persistent module 502) can be configured to execute a process, sub-process, thread, and / or service managed by an audio service 522 (e.g., instructions executed by persistent module 502 can run within audio service 522), which may run on one or more computer systems. In some embodiments, audio service 522 can be a process, which may run within a runtime environment, and instructions executed by a module (e.g., persistent module 502) can be a component of audio service 522 (e.g., instructions executed by persistent module 502 can be a sub-process of audio service 522). In some embodiments, audio service 522 can be a sub-process of a parent process. The instructions performed by a module (e.g., persistent module 502) may include processes, sub-processes, threads, and / or services performed by one or more components (e.g., localization status sub-module 506, acoustic data sub-module 508, and / or audio model sub-module 510). In some embodiments, the instructions performed by a module (e.g., persistent module 502) may be launched as a sub-process of audio services 522 and / or as a separate process in a different location than other components of audio services 522.For example, instructions executed by a module (e.g., persistent module 502) may run in a general-purpose processor, and one or more other components of audio services 522 may run in an audio-specific processor (e.g., DSP). In some embodiments, instructions executed by a module (e.g., persistent module 502) may run in a different process address space and / or memory space than other components of audio services 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may run as one or more threads within audio services 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may be instantiated within audio services 522. In some embodiments, instructions executed by a module (e.g., persistent module 502) may share process address and / or memory space with other components of audio services 522.

[0073] In some embodiments, the persistent module 502 may include a localization status submodule 506. The localization status submodule 506 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the localization status submodule 506 may be sub-processes of the persistent module 502. In some embodiments, the localization status submodule 506 may indicate whether localization has been achieved (e.g., the localization status submodule 506 may indicate whether the MR system has identified the real environment and / or located itself within the real environment). In some embodiments, the localization status submodule 506 may interface with a localization system (e.g., via an API). The localization system may determine a location for the MR system (and / or a user using the MR system). In some embodiments, the localization system may utilize techniques such as SLAM to create a three-dimensional model of the real environment and estimate the location of the system (and / or the user) within the environment. In some embodiments, the localization system can rely on a passable world system (described in more detail below) and one or more sensors (e.g., of the MR system 112, 200) to estimate the location of the MR system (and / or the user) within the environment. In some embodiments, the localization status submodule 506 can query the localization system to determine whether localization has currently been achieved for the MR system. Similarly, the localization system may inform the localization status submodule 506 of the success of the localization. The localization status (e.g., of the MR system 112, 200) may be used to determine whether the audio model should be updated (e.g., because the user's real-world environment has changed).

[0074] In some embodiments, the persistent module 502 may include an acoustic data sub-module 508. The acoustic data sub-module 508 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the acoustic data sub-module 508 may be sub-processes of the persistent module 502. In some embodiments, the acoustic data sub-module 508 may store one or more data structures representing acoustic data that may be used to create an audio model. In some embodiments, the acoustic data sub-module 508 may interface with a passable world system (e.g., via an API). The passable world system may include information about known real environments (e.g., rooms, buildings, and / or exterior spaces) and associated real and / or virtual objects. In some embodiments, the passable world system may include persistent coordinate frames and / or anchor points. The persistent coordinate frames and / or anchor points may be fixed points in space that may be known to the MR system (e.g., by a unique identifier). Virtual objects may be positioned relative to one or more persistent coordinate frames and / or anchor points to enable object persistence (e.g., a virtual object may appear to remain in the same location in the real environment regardless of who views the virtual object and regardless of any movements of the user). Persistent coordinate frames and / or anchor points may be advantageous, especially when two or more users with separate MR systems utilize different world coordinate frames (e.g., each user's location is specified as the origin with respect to its individual world coordinate frame). Object persistence across users can be achieved by transforming between individual world coordinate frames and a universal persistent coordinate frame and placing / referencing virtual objects relative to the persistent coordinate frame.In some embodiments, the passable world system can manage and maintain the persistent coordinate frames, for example, by mapping new areas and creating new persistent coordinate frames; by remapping known areas and reconciling new persistent coordinate frames with previously determined persistent coordinate frames; and / or by associating identifiable information (e.g., locations and / or nearby objects) with persistent coordinate frames. In some embodiments, the acoustic data sub-module 508 can query a separate system (e.g., the passable world system) for one or more persistent coordinate frames. In some embodiments, the acoustic data sub-module 508 can retrieve, facilitate access and management of associated persistent coordinate frames (e.g., persistent coordinate frames within a threshold radius of the user's location), including creation, modification, and / or deletion of associated acoustic data.

[0075] In some embodiments, the acoustic data stored in the acoustic data sub-module 508 can be organized into physically correlated modular units (e.g., a room may be represented by a modular unit and a chair in the room may be represented by another modular unit). For example, the modular units may include physical and / or perceptually relevant properties of a physical environment (e.g., a room). The physical and / or perceptually relevant properties may include properties that may affect the acoustic characteristics of the room (e.g., the dimensions and / or shape of the room). In some embodiments, the physical and / or perceptually relevant properties may include functional and / or behavioral properties, which may be interpreted by the rendering engine (e.g., whether a source outside the room should be occluded). In some embodiments, the physical and / or perceptually relevant properties may include properties of known and / or recognized objects. For example, the geometry of fixed objects (e.g., floors, walls, furniture, etc.) and / or movable objects (e.g., a mug) may be stored as physical and / or perceptually relevant properties and associated with a particular environment. In some embodiments, the physical and / or perceptually relevant properties may include transmission losses, scattering coefficients, and / or absorption coefficients. In some embodiments, the modular units may include physical and / or perceptually relevant connections (e.g., between other modular units or within modular units). For example, the physical and / or perceptually relevant connections may link two or more rooms together and describe how the rooms may interact with each other (e.g., cross-coupling gain levels between digital reverberators simulating the rooms and / or line-of-sight paths between two spaces).

[0076] In some embodiments, the physical and / or perceptually relevant properties may include acoustic properties such as reverberation time, reverberation delay, and / or reverberation gain. Reverberation time may include the length of time required for sound to attenuate a certain amount (e.g., 60 decibels). Sound attenuation may be the result of sound reflecting off surfaces in the real environment (e.g., walls, floors, ceilings, etc.) while losing energy due to sound absorption by, for example, room boundaries (e.g., walls, floors, ceilings, etc.), objects inside the room (e.g., chairs, furniture, people, etc.), and the air in the room. Reverberation time may be affected by environmental factors. For example, absorptive surfaces (e.g., cushions) may absorb sound in addition to geometric diffusion, and reverberation time may be reduced as a result. In some embodiments, it may not be necessary to have information about the original source to estimate the reverberation time of the environment. The reverberation gain may include a ratio of the direct / source / original energy of a sound to the reverberation energy of a sound where the listener and the source are substantially co-located (e.g., the energy of the reverberation resulting from the direct / source / original sound) (e.g., a user may clap their hands, producing a source sound that may be considered to be substantially co-located with one or more microphones mounted on a head-mounted MR system). For example, an impulse (e.g., a clap) may have an energy associated with the impulse, and a reverberation sound from the impulse may have an energy associated with the reverberation of the impulse. The ratio of the original / source energy to the reverberation energy may be the reverberation gain. The reverberation gain of a real environment may be affected, for example, by absorptive surfaces that may absorb sound, thereby reducing the reverberation energy.

[0077] In some embodiments, the acoustic data may include metadata (e.g., metadata of a physical and / or perceptual related nature). For example, information about the time and / or location where the acoustic data was collected may be included within the acoustic data. In some embodiments, confidence data associated with the acoustic data (e.g., estimated measurement accuracy and / or number of repeated measurements) may be included as metadata. In some embodiments, the type of modular unit (e.g., modular unit with respect to a room or linkage between modular units) and / or data versioning may be included as metadata. In some embodiments, unique identifiers associated with the acoustic data, persistent coordinate frames, and / or anchor points may be included as metadata. In some embodiments, persistent coordinate frames and / or relative transformations from anchor points and associated virtual objects may be included as metadata. In some embodiments, the metadata may be stored as a single bundle along with the acoustic data.

[0078] In some embodiments, the acoustic data may be organized by persistent coordinate frames and / or anchor points, which may be organized into a map. In some embodiments, the audio model may consider acoustic data organized by persistent coordinate frames and / or anchor points, which may correspond to locations in the environment. In some embodiments, acoustic data may be loaded into the acoustic data submodule 508 depending on the success of a localization event (which may be indicated by the localization status submodule 506). In some embodiments, all available acoustic data may be loaded into the acoustic data submodule 508. In some embodiments, only relevant acoustic data may be loaded into the acoustic data submodule 508 (e.g., acoustic data for persistent coordinate frames and / or anchor points within a certain distance of the location of the MR system).

[0079] In some embodiments, the acoustic data may include different states that may change according to changes in the real environment. For example, acoustic data for a given modular unit, which may represent a room, may include acoustic data for the room in an empty state and acoustic data for the room in an occupied state. In some embodiments, changes in the furniture arrangement of the room may be reflected by a change of state in the acoustic data associated with the room. In some embodiments, a modular unit (e.g., representing a room) may include different acoustic data for states when a door is opened or closed. The states may be represented as binary values ​​(e.g., 0 or 1) or continuous values ​​(e.g., the degree to which a door is open, the degree to which a room is occupied, etc.).

[0080] In some embodiments, the persistent module 502 may include an audio model submodule 510. The audio model submodule 510 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the audio model submodule 510 may include one or more data structures that represent an audio model for a real and / or virtual environment. In some embodiments, the audio model may be generated, at least in part, by the acoustic data stored in the acoustic data submodule 508. The audio model may represent the way a sound behaves in a particular environment. For example, a virtual sound generated by an MR system (e.g., the MR system 112, 200) may be modified by the audio model in the audio model submodule 510 to reflect the acoustic properties of the environment. A virtual concert presented to a user seated in a spacious concert hall may have similar acoustic properties as a real concert presented in the same concert hall. The MR system may locate itself relative to the concert hall, load the relevant acoustic data, generate the audio model, and model the acoustic properties of the concert hall.

[0081] In some embodiments, the audio model may be used to model sound propagation in the environment. For example, propagation effects may include occlusion, obstacles, early reflections, diffraction, time-of-flight delay, Doppler effect, and other effects. In some embodiments, the audio model may take into account frequency-dependent absorption rates and / or transmission losses (e.g., based on the acoustic data loaded into the acoustic data sub-module 508). In some embodiments, the audio model stored in the audio model sub-module 510 may inform other aspects of the audio engine. For example, the audio model may use the acoustic data to synthesize audio in the process (e.g., collisions between virtual and / or real objects).

[0082] In some embodiments, the audio rendering service 522 may include a rendering track module 514. The rendering track module 514 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the rendering track module 514 may include audio information that may then be presented to a user. In some embodiments, the MR system may present a virtual sound that includes several sound sources that are mixed together (e.g., a sound source of two swords clashing and a sound source of a person screaming). The rendering track module 514 may store one or more tracks that may be mixed with other tracks for presentation to a user. In some embodiments, the rendering track module 514 may include information about spatial sources. For example, the rendering track module 514 may include information about where the sound sources are located, which may be taken into account in the audio model and / or rendering algorithm. In some embodiments, the rendering track module 514 may include information about the relationship between modular units and / or sound sources. For example, one or more rendering tracks and / or audio models may be associated together as a single group.

[0083] In some embodiments, the audio rendering service 522 may include a location manager module 516. The location manager module 516 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the location manager module 516 may manage location information related to the audio engine (e.g., the current location of the MR system in the real environment). In some embodiments, the location manager module 516 may include a perception wrapper sub-module. The perception wrapper sub-module may be a wrapper around the perception data (e.g., what the MR system has detected or is detecting). In some embodiments, the perception wrapper may interface and / or translate between the perception data and the location manager module 516. In some embodiments, the location manager module 516 may include a head pose sub-module, which may include head pose data. The head pose data may include the location and / or orientation of the MR system (or the corresponding user) in the real environment. In some embodiments, the head pose may be determined based on the perception data.

[0084] In some embodiments, the audio rendering service 522 may include an audio model module 518. The audio model module 518 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the audio model module 518 may include an audio model, which may be the same audio model included in the audio model sub-module 510. In some embodiments, the modules 510 and 518 may maintain duplicate copies of the same audio model. For example, when an audio model is updated but sounds are to be presented to a user, it may be advantageous to maintain more than one copy of the audio model. It may be advantageous to update a copy of the model when the model is currently in use and then update the outdated model when the outdated model is unavailable (e.g., no longer in use). In some embodiments, the audio model may be transferred between modules 510 and 518 through serialization. The audio model in module 510 may be serialized and deserialized to facilitate data transfer to module 518. Serialization can facilitate data transfer between processors (eg, a general processor and an audio-specific processor) so that typed memory does not need to be shared.

[0085] In some embodiments, the audio rendering service 522 may include a rendering algorithm module 520. The rendering algorithm module 520 may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the rendering algorithm module 520 may include an algorithm for rendering a virtual sound so that it may be presented to a user (e.g., via one or more speakers of an MR system). The rendering algorithm module 520 may take into account an audio model of a specific environment (e.g., the audio model in modules 510 and / or 518).

[0086] In some embodiments, the audio service 522 can be a process, sub-process, thread, and / or service running on one or more computer systems (e.g., within the MR system 112, 200). In some embodiments, a separate system (e.g., a third party application) may request that an audio signal be presented (e.g., via one or more speakers of the MR system 112, 200). Such a request may take any suitable form. In some embodiments, a request that an audio signal be presented can include software instructions for presenting the audio signal, and in some embodiments, such a request may be hardware driven. The request may be issued with or without user involvement. Furthermore, such a request may be received via local hardware (e.g., from the MR system itself), via external hardware (e.g., a separate computer system in communication with the MR system), via the Internet (e.g., via a cloud server), or via any other suitable source or combination of sources. In some embodiments, audio service 522 may receive the request, render the requested audio signal (e.g., through rendering algorithm 520, which may consider the audio model from blocks 510 and / or 518), and present the requested audio signal to the user. In some embodiments, audio service 522 may be a process that runs continuously (e.g., in the background) while the operating system of the MR system is running. In some embodiments, audio service 522 may be an instantiation of a parent background service, which may act as a host process for one or more background processes and / or sub-processes. In some embodiments, audio service 522 may be part of the operating system of the MR system. In some embodiments, audio service 522 may be accessible to applications that may be launched on the MR system.In some embodiments, a user of the MR system may not directly provide input to audio services 522. For example, a user may provide input (e.g., movement commands) to an application (e.g., a role-playing game) running on the MR system. The application may provide input to audio services 522 (e.g., to render footstep sounds), and audio services 522 may provide output (e.g., rendered footstep sounds) to the user (e.g., via speakers) and / or to other processes and / or services.

[0087] 6 illustrates an example process for updating an audio model, according to some embodiments. In step 606, localization may be determined (e.g., the MR system may successfully identify its location within the environment). In step 607, which may occur within persistent module 602 (which may correspond to persistent module 502), a notification of successful localization may be issued. In some embodiments, the notification of successful localization may trigger a process to update the audio model (e.g., because the previous audio model may no longer apply to the current location).

[0088] In step 608, it may be determined whether to initiate a call for the acoustic data. It may be desirable to set one or more conditions for initiating the call so that the audio model is not updated too frequently. For example, if a user using the MR system moves only slightly around the room, it may not be desirable to update the audio model (e.g., because the updated model may not be perceptually distinguishable from the existing model and / or it may be computationally expensive to continually update the model). In some embodiments, the threshold condition in step 608 may be based on time. For example, the call may be initiated only if the call has not already been initiated 5 seconds prior. In some embodiments, the threshold condition in step 608 may be based on location. For example, the call may be initiated only if the user has changed location a threshold distance amount. Note that other threshold conditions may be used as well. In some embodiments, steps 607 and / or 608 may occur within a location location status submodule (e.g., location location status submodule 506).

[0089] If it is determined that a call should be initiated, the persistent coordinate frames may be retrieved in step 610. In some embodiments, only a subset of the available persistent coordinate frames may be retrieved in step 610. For example, only persistent coordinate frames near the location may be retrieved.

[0090] In step 612, acoustic data may be read. In some embodiments, the acoustic data read in step 612 may correspond to acoustic data stored in the acoustic data sub-module 508. In some embodiments, only a subset of the available acoustic data may be read in step 612. For example, acoustic data associated with one or more persistent coordinate frames and / or anchor points may be read. In some embodiments, steps 612 and / or 614 may occur within an acoustic data sub-module (e.g., acoustic data sub-module 508).

[0091] In step 614, an audio model may be constructed and / or modified. In some embodiments, the audio model may take into account the acoustic data retrieved in step 612, and the audio model may model the acoustic characteristics of a particular environment. In some embodiments, step 614 may occur within an audio model sub-module (e.g., audio model sub-module 510).

[0092] In step 616, it may be determined whether the copy of the audio model should be updated. To avoid interrupting a service (e.g., presenting audio to a user), it may be desirable to set one or more conditions for updating the copy of the audio model. For example, one condition may evaluate whether a copy of the audio model exists (e.g., within the audio rendering service 604 but outside the persistent module 602). If a copy of the audio model does not exist, the audio model may be retrieved by the audio rendering service 604 (which may correspond to the audio rendering service 522). In some embodiments, the condition may evaluate whether the copy of the audio model is currently in use (e.g., whether the audio model is being used to render and present audio to a user). If the copy of the audio model is not in use, the updated audio model (e.g., the audio model generated in step 614) may be propagated to the copy.

[0093] In step 618, the audio rendering service 604 may retrieve a copy of the audio model (e.g., the audio model generated in step 614). The audio model may be transferred using data serialization and / or deserialization.

[0094] In step 620, outdated audio models may be optionally deleted and / or disabled. For example, the audio rendering service 604 may have a first existing audio model that was previously used. The audio rendering service 604 may retrieve a second updated audio model (e.g., from the persistent module 602) and delete and / or disable the first existing audio model.

[0095] A notification may be issued regarding the new audio model at step 622. In some embodiments, the notification may include a callback function to a client (e.g., a third party application) that can subscribe to be notified when the audio model changes.

[0096] 7 illustrates an exemplary process for updating an audio model, according to some embodiments. In step 706, audio data may be received (e.g., via one or more sensors of the MR system 112, 200). In some embodiments, the audio data may be manually entered (e.g., a user and / or developer may manually enter reverberation time, reverberation delay, reverberation gain, etc.). In step 708, which may occur within the persistent module 702 (which may correspond to the persistent module 502), an associated environment may be identified. The associated environment may be identified by metadata that may accompany the audio data (e.g., the metadata may carry information about one or more persistent coordinate frames and / or anchor points that may be known to the MR system).

[0097] In step 710, it may be determined whether the associated environment is new. For example, if the associated environment cannot be identified and / or is associated with an unknown identifier, it may be determined that the audio data is associated with a new environment. If it is determined that the associated environment is not new, the copy of the authoritative audio model may be updated (e.g., with room properties that may be derived from the audio data). If it is determined that the associated environment is new, the new environment may be added to the copy of the authoritative audio model, and the copy audio model may be updated accordingly. In some embodiments, the new environment may be represented by a new modular unit.

[0098] In step 716, metadata associated with the new environment (e.g., metadata associated with the new modular unit) may be initialized. For example, metadata associated with measurement counts, confidence levels, or other information may be created and bundled with the new modular unit.

[0099] At step 718, the authoritative audio model in the persistent module 702 may be updated. For example, the authoritative audio model may be duplicated from the updated copy audio model. In some embodiments, the authoritative audio model may be locked at step 718 to prevent further changes from being made to the authoritative audio model. In some embodiments, changes may still be made to a copy of the authoritative audio model (which may still reside in the persistent module 702) while the authoritative audio model is locked.

[0100] In step 720, the acoustic data associated with the new audio data may be stored. For example, a new modular unit associated with a new room may be stored and / or passed to the passable world system (which may be made accessible to the MR system in the future, if desired). In some embodiments, step 720 may occur within the acoustic data sub-module 508. In some embodiments, step 720 may occur sequentially, following step 718. In some embodiments, step 720 may occur at an independent time as determined by other components, for example, based on availability of the passable world system in which the acoustic data may be stored.

[0101] In step 722, the audio rendering service 704 (which may correspond to the audio rendering service 522) may retrieve a copy of the audio model. In some embodiments, the copy of the audio model may be the same audio model that was updated in step 718. In some embodiments, the data transfer may occur through serialization and deserialization. In some embodiments, the audio model may be locked while the serialization process is being performed, which may prevent the audio model from being modified while the snapshot of the audio model is being created.

[0102] In step 724, copies of the outdated audio models may be deleted and / or invalidated.

[0103] A notification may be issued regarding the new audio model in step 726. The notification can be a callback function to clients that have subscribed to be notified when the model is updated.

[0104] In step 728, the authoritative audio model may be released, which may indicate that the serialized bundle corresponding to the audio model may be deleted. In some embodiments, it may be desirable to lock the authoritative audio model in the persistent module 702 while a copy of the authoritative audio model is being transferred to the audio rendering service 704. It may be desirable to unlock the authoritative audio model once the audio rendering service 704 has finished reading its copy of the authoritative audio model so that the authoritative audio model may continue to be updated.

[0105] In some embodiments, an audio rendering service (e.g., audio rendering service 522) may not manage the interaction between a persistent module (e.g., persistent module 502) and a rendering algorithm (e.g., rendering algorithm module 520). For example, the rendering algorithm 520 may communicate directly with the persistent module 502 to retrieve an updated audio model. In some embodiments, the rendering algorithm 520 may include its own copy of the audio model. In some embodiments, the rendering algorithm 520 may access the audio model in the persistent module 502.

[0106] As provided above, disclosed herein are systems and methods for storing, organizing, and maintaining audio data for a mixed reality system.

[0107] The system may include one or more sensors of a head-worn device, a speaker of the head-worn device, and one or more processors configured to execute a method. The method for execution by the one or more processors may include receiving a request to present an audio signal, identifying an environment via the one or more sensors of the head-worn device, retrieving one or more audio model components associated with the environment, generating a first audio model based on the audio model components, generating a second audio model based on the first audio model, determining a modified audio signal based on the second audio model and based on the request to present the audio signal, and presenting the modified audio signal via the speaker of the head-worn device.

[0108] In some system aspects, the second audio model may be generated by an audio service. In some system aspects, the modified audio signal may be determined by an audio service. In some system aspects, the second audio model may be a replica of the first audio model.

[0109] In some system aspects, the one or more audio model components may include one or more dimensions of the environment. In some system aspects, the one or more audio model components may include reverberation time. In some system aspects, the one or more audio model components may include reverberation gain. In some system aspects, the one or more audio model components may include a transmission loss coefficient. In some system aspects, the one or more audio model components may include an absorption coefficient.

[0110] The method may include receiving a request to present an audio signal. An environment may be identified via one or more sensors of the head-worn device. One or more audio model components associated with the environment may be retrieved. A first audio model may be generated based on the audio model components. A second audio model may be generated based on the first audio model. A modified audio signal may be determined based on the second audio model and based on the request to present the audio signal. The modified audio signal may be presented via a speaker of the head-worn device.

[0111] In some method aspects, the second audio model may be generated by an audio service. In some method aspects, the modified audio signal may be determined by an audio service. In some method aspects, the second audio model may be a replica of the first audio model.

[0112] In some method aspects, the one or more audio model components may include one or more dimensions of the environment. In some method aspects, the one or more audio model components may include a reverberation time. In some method aspects, the one or more audio model components may include a reverberation gain. In some method aspects, the one or more audio model components may include a transmission loss coefficient. In some method aspects, the one or more audio model components may include an absorption coefficient.

[0113] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method for execution by the one or more processors may include receiving a request to present an audio signal, identifying an environment via one or more sensors of a head-worn device, retrieving one or more audio model components associated with the environment, generating a first audio model based on the audio model components, generating a second audio model based on the first audio model, determining a modified audio signal based on the second audio model and based on the request to present the audio signal, and presenting the modified audio signal via a speaker of the head-worn device.

[0114] In some non-transitory computer readable media aspects, the second audio model may be generated by an audio service. In some non-transitory computer readable media aspects, the modified audio signal may be determined by an audio service. In some non-transitory computer readable media aspects, the second audio model may be a replica of the first audio model.

[0115] In some non-transient computer readable media aspects, the one or more audio model components may include one or more dimensions of the environment. In some non-transient computer readable media aspects, the one or more audio model components may include a reverberation time. In some non-transient computer readable media aspects, the one or more audio model components may include a reverberation gain. In some non-transient computer readable media aspects, the one or more audio model components may include a transmission loss coefficient. In some non-transient computer readable media aspects, the one or more audio model components may include an absorption coefficient.

[0116] The system may include one or more sensors of a head-mounted device, a speaker of the head-mounted device, and one or more processors configured to execute the method. The method for execution by the one or more processors may include receiving audio data via one or more sensors of the head-mounted wearable device, determining one or more acoustic characteristics of an environment based on the audio data, determining an associated environment for the audio data, generating an updated audio model based on the one or more acoustic characteristics of the environment, retrieving the updated audio model, generating a notification associated with the updated audio model, and presenting audio via the speaker of the head-mounted device based on the signal updated audio model.

[0117] In some system aspects, the updated audio model may be retrieved by an audio service. In some system aspects, the notification may be generated by the audio service. In some system aspects, generating the updated audio model may be further based on a previous audio model.

[0118] In some system aspects, the one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some system aspects, the one or more acoustic characteristics of the environment may include reverberation time. In some system aspects, the one or more acoustic characteristics of the environment may include reverberation gain. In some system aspects, the one or more acoustic characteristics of the environment may include a transmission loss coefficient. In some system aspects, the one or more acoustic characteristics of the environment may include an absorption coefficient.

[0119] The method may include receiving audio data via one or more sensors of the head-worn device. One or more acoustic characteristics of the environment may be determined based on the audio data. An associated environment may be determined with respect to the audio data. An updated audio model may be generated based on the one or more acoustic characteristics of the environment. The updated audio model may be retrieved. A notification associated with the updated audio model may be generated. An audio signal based on the updated audio model may be presented via a speaker of the head-worn device.

[0120] In some method aspects, the updated audio model may be retrieved by an audio service. In some method aspects, the notification may be generated by the audio service. In some method aspects, generating the updated audio model may further be based on a previous audio model.

[0121] In some method aspects, the one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some method aspects, the one or more acoustic characteristics of the environment may include reverberation time. In some method aspects, the one or more acoustic characteristics of the environment may include reverberation gain. In some method aspects, the one or more acoustic characteristics of the environment may include a transmission loss coefficient. In some method aspects, the one or more acoustic characteristics of the environment may include an absorption coefficient.

[0122] A non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method. The method for execution by the one or more processors may include receiving audio data via one or more sensors of a head-worn device, determining one or more acoustic characteristics of an environment based on the audio data, determining an associated environment for the audio data, generating an updated audio model based on the one or more acoustic characteristics of the environment, retrieving the updated audio model, generating a notification associated with the updated audio model, and presenting an audio signal based on the updated audio model via a speaker of the head-worn device.

[0123] In some non-transient computer readable media aspects, the updated audio model may be retrieved by an audio service. In some non-transient computer readable media aspects, the notification may be generated by the audio service. In some non-transient computer readable media aspects, generating the updated audio model may be further based on a previous audio model.

[0124] In some non-transient computer readable media aspects, the one or more acoustic characteristics of the environment may include one or more dimensions of the environment. In some non-transient computer readable media aspects, the one or more acoustic characteristics of the environment may include a reverberation time. In some non-transient computer readable media aspects, the one or more acoustic characteristics of the environment may include a reverberation gain. In some non-transient computer readable media aspects, the one or more acoustic characteristics of the environment may include a transmission loss coefficient. In some non-transient computer readable media aspects, the one or more acoustic characteristics of the environment may include an absorption coefficient.

[0125] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications should be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. A system comprising: one or more sensors in a head-worn device; a speaker of the head-worn device; and one or more processors configured to execute the method; Equipped with The method comprises: determining a location of the head-worn device via the one or more sensors of the head-worn device; determining whether a difference between a location of the head worn device and a location associated with an audio model exceeds a threshold; upon determining that the difference exceeds the threshold, determining one or more acoustic characteristics of an environment of the head-worn device based on audio data, the audio data being received via the one or more sensors of the head-worn device; generating an updated audio model based on the one or more acoustic characteristics of the environment; and presenting a first audio signal based on the updated audio model via the speaker of the head-worn device. and pursuant to a determination that the difference does not exceed the threshold value; presenting a second audio signal based on the audio model via the speaker of the head-worn device; Including, the system.

2. The system of claim 1, wherein the method further includes receiving the audio model at the head-worn device via an audio service.

3. The system of claim 1, wherein the method further includes generating a notification associated with the updated audio model via an audio service.

4. The system of claim 1, wherein generating the updated audio model includes generating the updated audio model based on a previous audio model.

5. The system described in claim 1, wherein the one or more acoustic properties of the environment include one or more dimensions of the environment.

6. The system of claim 1, wherein the one or more acoustic characteristics of the environment include reverberation time.

7. The system of claim 1, wherein the one or more acoustic characteristics of the environment include reverberation gain.

8. The system described in claim 1, wherein the one or more acoustic properties of the environment include a transmission loss coefficient.

9. The system of claim 1, wherein the one or more acoustic properties of the environment comprise an absorption coefficient. determining a location of a head-worn device via one or more sensors of the head-worn device; determining whether a difference between a location of the head worn device and a location associated with an audio model exceeds a threshold; upon determining that the difference exceeds the threshold, determining one or more acoustic characteristics of an environment of the head-worn device based on audio data, the audio data being received via the one or more sensors of the head-worn device; generating an updated audio model based on the one or more acoustic characteristics of the environment; and Presenting a first audio signal based on the updated audio model via a speaker of the head-worn device. and pursuant to a determination that the difference does not exceed the threshold value; presenting a second audio signal based on the audio model via the speaker of the head-worn device; A method comprising:

11. The method of claim 10, further comprising receiving the audio model at the head-worn device via an audio service.

12. The method of claim 10, further comprising generating, via an audio service, a notification associated with the updated audio model.

13. The method of claim 10, wherein generating the updated audio model includes generating the updated audio model based on a previous audio model.

14. The method of claim 10, wherein the one or more acoustic properties of the environment comprise one or more dimensions of the environment.

15. The method of claim 10, wherein the one or more acoustic properties of the environment include reverberation time.

16. The method of claim 10, wherein the one or more acoustic properties of the environment include reverberation gain.

17. The method of claim 10, wherein the one or more acoustic properties of the environment comprise a transmission loss coefficient.

18. The method of claim 10, wherein the one or more acoustic properties of the environment comprise an absorption coefficient.

19. A non-transitory computer-readable medium, the non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: determining a location of a head-worn device via one or more sensors of the head-worn device; determining whether a difference between a location of the head worn device and a location associated with an audio model exceeds a threshold; upon determining that the difference exceeds the threshold, determining one or more acoustic characteristics of an environment of the head-worn device based on audio data, the audio data being received via the one or more sensors of the head-worn device; generating an updated audio model based on the one or more acoustic characteristics of the environment; and Presenting a first audio signal based on the updated audio model via a speaker of the head-worn device. and pursuant to a determination that the difference does not exceed the threshold value; presenting a second audio signal based on the audio model via the speaker of the head-worn device; 16. A non-transitory computer readable medium comprising:

20. The non-transitory computer-readable medium of claim 19, wherein the method further includes receiving the audio model at the head-worn device via an audio service.

Images